EP96 Cloud Security Observability for Detection and Response

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday afternoon Pacific time, please do hit that subscribe button. Please rate our podcast if that's an option that you feel comfortable performing. You can follow the show and argue with your hosts on Twitter as well. And believe me, people do. twitter.com/cloudsecpodcast. Anton, we're talking about observability today. And I have a feeling we've got some observations about that, don't we?

Yes, we do. And I kind of started the episode, as you'll learn in a couple of minutes, from kind of hand-waving and saying that observability is a very hand-waving subject for a lot of people because they don't have the precise definition. And then we kind of crafted the definition. So, you'll see.

We'll see. This episode also features one of my favorite recommended readings we've had on the show. So far, it's not somebody citing the SRE book listeners, so that's good. This one is very different. This one is sci-fi, right?

Yes, correct. So this, we're kind of, it's almost like feels like a spoiler, but—

Can we spoil our own show?

Yeah, yeah, yeah. We'll, we'll do this. How about we tell the recommended reading now and we'll explain why at the end? Yeah. So a guest recommended Ray Bradbury. book, Marshall Chronicles. It's a very well-known sci-fi. I assumed it's from the '80s, but it's really more like from the '50s, actually. But why it's relevant for a cloud security podcast, and believe me, it is very relevant for a cloud security podcast.

We'll leave that for the end, listeners.

You will learn that at the very end.

Well, with that, let's welcome today's guest.

And today our guest is Jeff Bollinger, Director of Incident Response and Detection Engineering. That sounds really cool. At LinkedIn. Welcome, Jeff.

Thanks, Anton. Good to be here.

Without going back in what was the first time we met, whether it was in the mid-2000s or 2010s, I want to touch on the fun topic we have. And before we go there, I want to say that it was my obsession a little bit, which I was not able to satisfy. So today, I hope I would satisfy that obsession a little bit. The topic is observability and, of course, security use cases. So observability sounds cool, but relating to security, at least in my experience, has been very hand-wavy. People say, oh yeah, we're going to use it, but it's not really clear what they have in mind. So, I wanted to start here. What are the use cases? What's your view on observability data for security? And perhaps while you're doing it, define observability for us.

Yeah, absolutely. I think it's important to understand what that actually means because there are some kind of conflicting terms as well with observability, namely visibility. I think one that we've all relied on a lot in the last several years, but basically, It's a really old concept. The idea here is that if you can observe the outputs of a system, you can infer the functionality or how things are working properly with inside of a system itself, in particular, a complex distributed system, usually one that's self-regulating, right? So that's a pretty formal definition of it, but what does it mean for security? For me, what it actually means is when I say visibility, I'm able to say that I can see this bit of data, I can see this part of the network. But what observability gives me is it doesn't show me that something happened, but shows me context around how something happened, or even potentially why something happened. And for the context of IR, of course, those are critical pieces. That's ultimately what we're doing at the end of the day. But for me, that's the big key difference between visibility and observability, is that observability gives us the opportunity to probe a little bit deeper to understand how something might have failed or been compromised.

But you mentioned why as well, and why is always very tricky, whether you're doing traditional logs, you know, 1970s, 1980s syslog, or modern observability or tracing. Why is kind of opaque, right? So I'm sorry for going off track from our own track, but you mentioned why, and how does observability help us to know why?

Well, it probably is easier to discuss this in terms of real-world examples, right? Because it's a lot easier to formulate. So abstract here, but really conceptually what that means is if we had a complex system that was compromised by a threat actor, right, we might not be able to tell exactly where that happened, right? We may only observe that through the output from that system. They're serving up content we weren't expecting. They're sending out additional URLs. They're creating new websites with domains that are under control by the attacker, right? So I think with those bits of information, we can maybe figure out how that happened, but I think an astute developer might be able to even understand why it happened, right? Oh, it looks like they're pushing it out on this particular function here. I recall that maybe we had some weak input validation there, right? Or we had some— this looks like this could have only have come from this one part of our middleware because it's the only part that does URL rewrites, right? So I think you can get some clues. from the contextual analysis that you get from observability that may be able to tell you why. But I definitely agree, it's not gonna give you why every time. I think it really takes an astute developer to determine that.

Yes, and I think that's the trick. And perhaps that's a useful reminder to the audience as well, is that a person who understands the system upon seeing the observability data can often say why, but a random person who is given a printout of logs probably would not say why from the, you know, in-app telemetry or whatever else. One other thing I wanna pick on is, Some people define observability a lot more narrower. Like, they go for, like, application traces, in-app telemetry, but I felt like you went a bit broad. So any comments on this? Like, it feels like observability to me is kind of a peer to visibility in your world. It's not just, like, in-app telemetry.

Well, I think you have to start broad because it is such a complex topic, right? I mean, realistically, what you're looking at here is, especially with a cloud system, right? We have so many moving parts. A complex chain of interconnected, interdependent functions, services, applications, right, that are all together. So, yeah, observability requires the ability to be able to trace through that and kind of recreate that. And given that, you know, basically, I've been doing the IR thing, that's essentially what I know the best. When it comes to IR, right, in this regard, we have to basically be able to retrace what happened, build a timeline, show what occurred, and gather all the evidence that's possible. So with visibility, yes, visibility means we have syslog, right? Observability means we have syslog from this host that runs this application and they have this particular app log that always writes to this syslog facility, right? And we know that only this host does that. So getting down to that, you get more precise. So that's why I started off broad, but certainly you could look into it in terms of application tracing and that's ultimately where you would go, right? Where did this identity interact with this API, which interacted with the following downstream services, right? There's definitely a piece for it for that, but I think generically, Observability really just means the ability to get more information about what you're processing versus just that you knew something happened, right? It's not a health check, it's actually a deep dive.

Yeah, but it's also not just user such and such authenticated successfully. That to me is still in the traditional login domain and it doesn't tell me, well, anything except for that fact, right? Like user such and such authenticated successfully.

Right, exactly. Yep. But we don't know what was the context of that. Was this an API? Was this a service principal? Was this an account that we know about, right? Like all these different pieces are things that kind of come into question whenever there's an incident.

And of course, you mentioned incidents in IR, and of course, to a very large extent, that's your area of expertise. But I mentioned in your title that it's detection and response. And so are we applying this mostly for dig this stuff after it hits the fan, or are there more, you know, I hate the word proactive, I'm not gonna say proactive, more of a detection use case, like more detecting things as they happen, as opposed to digging through things after they happen?

Yeah, that's a really That's a good point because I feel like that kind of gets glossed over quite often, right? It's like we have logs, so we're good. Well, did you actually do something with those logs, right? And also these logs are really expensive. Should we really be, you know, indexing this terabyte of data per day, right? Is it really making sense to us? I think that there's always a place in that for IR, right? Because you always wanna know what happened after the fact. So there'll always be a reason to go back and dig through logs, whether that's cold storage, hot storage, however you decide to do it, there's always gonna be a need for that. But on the detection engineering side, absolutely. Because all detection engineering is when you boil it down is really Setting traps, looking for signals, right? And essentially trying to get a certain condition triggered. So if we can take those signals from our observability pipeline, then absolutely we could do it. So if we don't, if we expect to see a certain type of data being emitted or a certain format being emitted from our application, and that's somehow deviated, something's happened, right? And we might want to look into that. Maybe it was something in the build process, maybe it was something malicious, but the only way to know is to actually trigger on that and investigate into it.

So it sounds like maybe I'm generalizing not quite right, but it sounds like a lot of the detection use cases would ultimately be anomaly detection use cases, would be something seen for the first time. Something not quite right, something not as before, something not like for other systems. All the classic anomaly approaches would apply here for observability data.

Yeah. Unexpected output is like the core definition of observability. Yeah.

Yeah. It's what we do. It's what we live for kind of thing.

Okay.

So I want to switch gears a little bit to the cloud because, hey, Cloud Security Podcast. By the way, there's a story about why our podcast is called Cloud Security Podcast. And I'm not telling the story on the air because it involves some administrative decisions here. The idea is that in the cloud, some of the observability mechanisms are superior while some others are challenged. So why don't we look at the network side first? So what can we do in this era of pervasive encryption, TLS 1.3? So what can we do? What can we do better in the cloud in terms of network observability?

Yeah, it's a tough one for sure. But like anything, it really starts with a super strong foundation of really strong security architecture, right? And proper management planes, data planes, access planes, presentation planes, All that complicated stuff that goes into building your cloud application is a critical component here, right? Because without that, we really don't even know where to look. And when you think about traditional network monitoring, right, the terms we always used to use were choke point, right? North-south, whatever you want to call it, right?

Yes.

Where is the choke point in between 2 functions, right? Where's the north-south in between a database and middleware, right? You don't have these terms anymore. They're very antiquated. So in fact, some situations you may not even have access to layer 3 traffic. aspects of networking in the cloud, right? So absolutely. But that said, it's not any different than the problems that we have had before, right, with TLS. So in particular, all traffic being encrypted, in particular, the best signal that we could get from a TLS stream, at least in 1.2 and earlier, is the Client Hello, right? And a bunch of other components that we can use for profiling. We still could do some profiling, right, with TLS 1.3, even though that's encrypted, because we can still run things like JA3.

Yeah, but it's very shallow. Yeah, it can be done, and there are vendors who do it, but they're shallow.

But they're very shallow. Yeah. And you gotta get lucky, right? I really don't think—

Yeah.

You have to know what you're looking for in that particular regard. So with that, yeah, I think there's some issues there, but I will say I've always been a huge fan of NetFlow or IPFIX, whatever you wanna call it, right? And NetFlow doesn't give you a full packet, right? It doesn't show you the contents of a payload, but it does give you a lot of metadata, right? It shows you the phone call and how long it lasted. And I think those bits of metadata are useful for people to build context and to do abstractions from that, right? So we may not have a smoking gun that says, yes, there was absolutely a malicious payload in this TLS encrypted packet. But we do have bits of metadata that we can tie together to build a case, right? That's the bits of evidence we've gathered. We know that it came from this particular IP address. Okay, well, that was a proxy network that we have. Okay, well, that gives us some scoring, right? We also know that we've never seen this sort of encryption cipher used before. That's also unusual. 99% of browsers use this, right?

Yeah, that would worry me if it's my own cloud. Yeah. And I put stuff in there and then somebody's encrypting things with things I didn't put in there. I'd freak out. This is like a 3:00 AM wake up kind of caliber event.

Totally.

For, for a lot of people.

Yeah. And also, I should also say with networking, I mean, one of the other pieces that I think we tend to not think about a lot, at least in the context of cloud, is DNS, right? I mean, isn't the answer, it's always DNS, right? Is the answer, right? But when it comes to like observability and monitoring and building detections, right? I mean, you think about all the indicator feeds that people subscribe to or, or collect or, or input, right? A vast majority of the ones are, are DNS, right? And you can get that information from your cloud provider. So who you resolve for, what you resolve, You know, who resolves something, right? These are all bits of information that even though the rest of the data in that stream are gonna be encrypted, you still know some hostnames, right? You still have some indicators and pieces of information that you can use to build the story. So I don't think all is lost when we encrypt everything, but I also think that you have to think more creatively. One example I had was sort of a, sort of a friendly argument I had with some friends yesterday, which was, is an S3 access log, is that a network level component or is it a— Or storage level? Yeah. Yeah. Like a storage level or, or an access or an application log, right? And you think about it, someone connects to your bucket, I mean, they're connecting to it. There's an access log that showed who came there from what IP, whatever else, right? How is that any different than, you know, an Apache access log, right? It still shows the same thing. Now, I didn't necessarily have a pcap running on my web server, but my application told me that this IP connected, right? So if you put all these layers together, layer 7 all the way down to what you can get hold of in the cloud, I think there is enough context that can be built to discover suspicious events, if not outright malicious altogether.

But it ends up, this particular debate just shifts into the cocktail conversation slash beer conversation very quickly because it's like, is S3 bucket, is storage bucket more of a hard drive or more of an application that gives storage? So that means, is it a network access or is it an application access? I mean, I feel like I need a drink and coffee doesn't quite cut it. So we're already debating it because is it a cloud log, an application log, or a network log?

It's all 3. It's all of it.

It's all 3. Exactly. But then it also happens to be a very voluminous log. And if you're doing, you know, here people have a similar debate over, say, BQ, BigQuery logs, right? Like, would I tell you, hey dude, just collect all BigQuery access logs in one place? Like, it's going to blow your mind and the budget, right? Frankly. But are they useful? Sure, they're useful. But that's not the point here. And the same with storage buckets.

That's right. That's right. And really, that is, isn't that what, in its essence, what we're trying to do with detection engineering is trying to find usable, useful components in our data sources, right? Because we know that when there's no attacks, 100% of the logs are fine, right? But when there are attacks, there could be 1% in there that you have to find. And I think that's the tricky part is, again, do we keep stuff offline for investigations later, or do we try to get out in front of it and look for things that we know of? And I also think that in the cloud environment and the way that we're pushing out software and doing deployments, right? I think there's also a lot of opportunity there as well. Like, they're in the actual unit testing. Or acceptance testing, right? In those particular scenarios, there's opportunities there for checking for security components, right? And hey, we didn't expect this output. That's unusual. I think from there, we can also kind of go back a little bit in the process and get ourselves some telemetry before it gets encrypted, right? And I think that's something helpful as well.

Right. And I think that's the chance to insert, both insert visibility components and judge the visibility components, observability traces before you deploy. So again, with modern cloud stacks, you can do it. With old world, it's like you get whatever application logs they put in there, right? And if they're in German and written in the 1980s, sorry. Yeah, it's like I'm remembering some nightmares about SAP log analysis. And so, yeah, this is kind of the story. But it kind of makes me nervous also as we think about it, because as people deploy SaaS, more software as a service, and if you build applications for others that are software as a service, you don't really have old-style network monitoring or network observation, and you also don't have all that much of the endpoint observation. So where do you go? Like, sometimes people on social media get in those, you know, pretty vile debates over, I'm the EDR guy, you're the NDR guy, let's have a fight. But ultimately, in SaaS, both kind of have to die, right? Like, maybe it's a very dark view, but how do you approach the software-as-a-service land in this case?

In my experience, the only thing that's worked so far up to this point has been focusing on it, on the identity layer, right? Because really all you're doing there is, okay, this person who, who purports to be in my organization has now authenticated to this SaaS service, right? And now they're running reports in that SaaS service. Now, to be fair, a lot of the major SaaS platforms have tons of different options for securing access, right? Access control. Mm-hmm. They have various levels of system administrator, right? But when you think about that, right, what are they really saying? They don't really care if you're checking, you know, this tool from your phone or if you're looking at it from a personally owned laptop, or a public library, it's really about who do you say that you are when you connect to this, right? And what kind of conditional access do we have for that? So really that's the only way I think, because we've shoved all of our network traffic in, you know, in TLS or HTTPS, right? And put all of our DNS in there. We've put all of our TCP traffic now into a QUIC tunnel, right? So we can't see what's happening there.

So—

Yes. Intentionally things are being obfuscated for people for various reasons, some good, some not so good. But what that does is it forces us to be creative and look at the only level where we do have any kind of visibility, and that is the authentication level. Because someone at my organization has to be part of my organization's directory.

Wait a second. Sorry for interrupting, but you said identity first, and now you kind of switched a little bit and said authentication, but authentication is much narrower. So which one is it? And to me, that matters.

Identity is the broad area in which we're going to focus, right? But authentication is just a small piece of it, right? The bigger piece that tends to get glossed over is the authorization, the second A in AAA, right? That's the one that is the most important. And that's really once I've authenticated to whatever service I have, What do I have access to, right? That's really the core piece here. Okay, so if I, you know, if my business role enables me to log into this particular SaaS app, I should be able to do it. But if I'm not part of that business role, should everyone in the company be able to log in and have the same level of privileges? Probably not, right?

Yeah.

Is that going to lead to potential abuses? Possibly. If somebody has an endpoint that's compromised, right, for whatever reason, whether it's corporate-owned or not corporate-owned, right, and that entity intends to authenticate to the SaaS service and then authorize to various services in there, If there's no additional policies that control that, then they're going to have ultimate access into there, right? So I think that's the big piece here is like, yes, arguably that is a broader area, is identity. But really what we're focusing on is what do you have access to once you're authenticated? And also when you're authenticated, is it actually you or is it somebody else?

Yes. And again, Tim and me try not to play too much of the Zero Trust drinking games because ultimately, well, for good reasons. But in this case, you sort of, you do fit in the, in kind of ZT-style access. which provides logs in terms of who got access where, what did they do there. So it's sort of like if you marry these zero-trust access principles with that type of telemetry, you will get useful data even without EDR and good traffic capture, right?

And if you take a step back, right, and I hate to be so didactic here, right, but it always goes back to the threat modeling, right? That's what it always goes back to in this particular case. And so if you're worried about someone logging in and making changes to your payroll system, that's Sounds like an insider threat issue, right? And I think those are very difficult and have a whole different class of problems, right? But if you're dealing with opportunistic attackers, which let's face it, the vast majority of what people are out there dealing with, they're looking at people spinning up new services, people creating new accounts, right? These are sort of like sort of the basic anomalies that are pretty easy to determine, right? Like we didn't have these 5,000 accounts last week. Now we do. Which account created that? Oh, it was this account, right? So I think that's kind of the area there that we have to look into is like, You just have to get crafty in terms of where to look. And I think network and EDR are becoming less and less of a possibility for us. The only thing that's left is the identity space at this point. Yeah.

But I want to sort of gently prod at this because it was a conclusion that I've made and a lot of other people have reached the same conclusion that ultimately the battle, the DNR battles would be fought at the identity layer and that's all there is to it. But if you have a chance, if you're building your own app that for somebody else looks like SaaS, but for you it looks like an app you're building, And I'm not trying to hint that it's necessarily like your employer's app or frankly my employer's app. The point is that if you're sitting on the— if you're building an app, you can insert things in the app that would make observation detecting threats better. And you would have in-app telemetry, which isn't only about authentication. Like, wouldn't you want to go there as well? Wouldn't you want to go and put lots of things to output activity traces that aren't necessarily identity, but more inside the app activity? Wouldn't you go there as well?

Yeah, I definitely think that is reasonable depending upon the app itself.

Not for a SaaS user, but for a SaaS builder.

Yeah, for the SaaS builder, of course, because that gives them the telemetry that they can use to ensure that their clients and customers feel better about it. Like, yeah, we saw this, we detected it. I also should say that a large majority of the major SaaS platforms have their own IR teams dedicated to these platforms, right? So you do get some of that, and they have instrumented that. And in fact, I think largely where a lot of the really interesting work's happening in machine learning models are really on these platform apps where they're looking at thousands and thousands and millions of queries per second, right? And kind of trying to sort those out and figure that out. You can't do that without having some sort of tracing inside of that. But I do think that we're blurring the lines a little bit here because we can talk about that from a security InfoSec perspective or from the actual systems perspective, right? The availability. So yeah, arguably confidentiality, integrity, availability. Availability is a big part of both, right? But I do think that it's just a matter of how far you're willing to go for the monitoring. But also on the other side, I do think that Instrumenting at the app layer is gonna give you a lot more stats that your developers are going to need and probably aren't gonna be the top of mind for what your security staff is gonna be doing unless they're dedicated to monitoring that app exclusively. That app.

Yes. And this may be the juiciest line from the entire episode. Well, apart from other stuff, of course, because ultimately, if you are instrumenting the app for outputting telemetry, a vast majority of this would be used for operational purposes. But there would be some minority or there would be some use cases that will be used for security. But you're also telling me, which I agree with, that the person doing this analysis will have to be an application expert in this application. Otherwise, no luck.

I think so. Unless it's completely spoon-fed to you, I think it's really because of observability, because of how complicated things are and interdependencies, right? Also different code changes, right? I mean, people are constantly deploying, putting out new features. You have to be able to understand that. I think to understand, is this normal, expected, or is this just a new feature? So I definitely think there has to be some sort of context there in order to successfully do it.

Okay. No, that does make sense. And I think that's probably like, it's not necessarily an illusion because people who are showing up at your door and say, oh, just give me all the telemetry from inside your app and we'll just use it for DNR. And you'd be like, sorry, but you'd never figure it out. Not because you're dumb, but because you don't have some of the context you have and the team that's building the app has. And a lot of this context is not externalized anywhere.

Right. And in particular, I do think there's a slight difference. So I should say that I do think that there's the ability, if you have a tightly connected team, to actually to get some useful, usable data from those internal apps, right? I do think that's the case.

Yes.

But what I've run into problems in the past, like on the enterprise level, is, hey, here's a Tomcat debug, right? You know, it's 5,000 lines of this Java, you know, library entering memory. That doesn't really help me. I don't know that that's supposed to happen. I really don't know. I really don't care. Does that really show me anything? I think that's the difference there, is that if you do have a tight relationship, And a strong objective to actually do the monitoring, I think you're going to get more than just debug logs and something that's actually usable.

Right. But it's a lot trickier. And again, the other drinking game is, of course, saying DevSecOps or DevOpsSec. But in this case, it does imply that your teams deploying and building the app and teams securing the app are kind of the same team, and they're either married to each other or they're the same team, and then they can do it.

Yeah.

But if you have to email the frigging log to somebody, then who never met you, the chance of value, the chance of a detection of something useful is kind of low.

Yeah. And one thing that our team has done here to kind of get around that problem is we've developed what we call a security logging standard, right? And it's more than just a standard. It's also a guideline. It's also kind of a how-to. And that standard isn't just like, you know, basically long story short, what it shows us is here's what we want to be able to do as an incident response team. We want to be able to piece together an incident, build a timeline, and recreate what happened based on signal and telemetry from your application, right? And so we provide that and then give them kind of a lot of leeway to do so, right? And how that actually is done. So at the end of the day, as long as it's a human-readable output that we can parse and run against, I think it's okay. But in order to publish software at our company, you have to meet this pipeline, right? And part of the pipeline is, do you have logging? Does it meet the security logging standard? And if so, you get to deploy. If not, you need to go back to the drawing board and do whatever else. Now that said, I won't say that 100% of the time we get really awesome, beautiful JSON that we can look at, right? We don't always get that, but at least we're getting the idea out there. that not only should you care about building this in for your own purposes, but also if something goes wrong, we want to be able to help you from a security perspective as well. So that's been pretty helpful.

Do you guys plan to release or maybe show some of the maybe sanitized high-level version of this? Because I've seen some of these guides, frankly, before I was an analyst, I've written some of those for clients when it was like authentication, authorization changes, crashes, certain types of errors. Like at a high level, it feels kind of intuitive, but some of the stuff you can't give to a developer because you if developer is not familiar what constitutes a security-relevant change inside an app, that the log standard may not connect to him. So I'm really curious to see whether you have some kind of a clean sanitized version of that, maybe without details, because it may help the industry, frankly. Because some of these docs float around, but when they are high level, they sound like this: authentication, authorization, change, error, crash, resource issue. That's how mine sounded from 10 years ago. But a client saw the details, but the public saw only this list. And the list is kind of handy, but not hugely useful.

Yeah. I think it's a good idea. I definitely think it's somewhere there. I don't think there's anything in particular that's super unique to our environment that would make it not public worthy. But I think the question is, or the question that I always come up with, and I think you totally nailed it, which is that sometimes you can't give developers a really ill-defined requirement. They want to know precisely what is your story, right? What is your user story? How's this going to work? And you can't just say, I want to know how something happened, right? It's very difficult to do that. So there is some contextual components there, but there are some reusable bits that I think that can be broadcast across any application, any company. There's definitely some basic things that we've been following for a while, and I think definitely it's a worthwhile thing to put it out there.

Yes, and it probably won't just be authentication. I'm pretty sure there's maybe up to a dozen things that I can port. I would probably like— if more is better.

Yeah, some things are mandatory, right? Like authentication is obviously a mandatory piece. There's others that I would say are A nice to have, right? And if your app is fully compliant, you'll have all these. But I think you can, you can still deploy with just the basic requirements, but I would love to see all the rest, right? Otherwise, when something happens, we're gonna be calling you up and saying, hey, what does this mean, right? So I think that it's in your best interest to give us the data.

Perfect.

So I want to ask— we have our traditional closing question, but I want to ask one more question if we have time. So sometimes the app is just complex. It's, as you mentioned, it's distributed, it's probably spread over multiple clouds, and components and written by many people, sometimes hundreds of people. So any tips on dealing with excessively complex apps? How do you decipher or how do you do DNR on an excessively complex app of that sort?

Yeah, that is—

Your answer can be excessively complex.

That's fine.

Yeah.

Well, okay. Let me just— yeah, that is a great question. And it's tough, right? Because you're totally right, especially that scenario you highlighted, right? Where it's going across maybe your own data center, multiple clouds, right? I think it does become really, really difficult. That's why I try to get that bit in about the architecture up front, right, of a secure architecture. Because if I say, for example, if I know that I have a really solid control plane, right, and I know what the access list is for that. So here's how you get in there. Here's how you do things. Here's what you can do. That makes it a little bit easier for me to understand what the capabilities actually are, right? So at least I can know where to start looking because all of our apps live in the same environment with these same control planes, management planes, whatever else. If I can understand that and how, what are the access controls between the various levels there, who has access to what, that does help me at least understand where to focus my efforts. The rest of it though, I think you can't just like run a traceroute or an Nmap, right?

Yeah.

And get what you want. There's so many moving parts there. And honestly, when you have things that are short-lived functions that are only up for a short period of time, it's really hard to know what actually happened. So what I would say is the advice I have is if you have a strong architecture, which I have to imagine a lot of folks do, at least understand where you could look for problems, right? We talked about choke points before. There are different areas you can look for, even if it's complicated. You don't necessarily have to know how everything worked if you saw Command injection was successful and a file was dropped, right? You start there and then work your way backwards. And honestly, the only way that I know how to do it is to do it that way. Start from the damage and work your way backwards. That tends to be the best way. But, you know, you can go into a black box pretty quickly with that approach as well too.

Yes, that's exactly right. That's my fear is that with the complex app, you may do this whole walking the path from damage and at some point you end up in a black box. It's like, okay, sorry, the answer is somewhere in this area, but there's no path leading there. Okay. This is a depressing note, which is perfect, I guess, for a security podcast. So our traditional closing question is just like, give the audience one tip on using observability for security or anything related, and probably give us some resources for further reading, your own blogs, anything.

Sure. Yeah.

You can say go to LinkedIn. That's fine too.

I strongly recommend LinkedIn, but I would also say that one tip that I wanted to put out here for observability, and because you asked about the networking bit, I wanted to highlight this. Something that's worked in the past really, really well, and I would encourage folks to look into this, is often an organization will be granted a series of netblocks, right? Or a couple ASNs, however many you have, whatever else. You can also have the same thing granted in your cloud services as well, right? In these environments, one of the things that we've found to be super valuable is if you have a subnet or an ASN that's what we used to call darknet, right? It's unallocated. No one's using it. No service is there. If there are attempted scans, connections, probes to that network, you know that that is not expected, right? It's an easy one, right? So having a reserved set of dark space that you can look for connections to that should never ever happen is usually a really nice leading indicator that something bad is happening, in particular if it's coming from your network, right? But if it's coming from externally, it's a different story. But if it's coming from your network internally, someone is probing, looking to move laterally, right? So I would strongly recommend that from an observability perspective alone, just having some reserved stuff that You don't expect to be touched. When it does get touched, that's really weird. That's almost a really easy signal to do.

So, it's almost like this advice probably was first given, I don't know, 20 years ago, but it aged really well.

Yes.

I've learned that myself is that this is— people may make snide remarks about it because, hey, what is this, 1990s stuff? But, like, it works today in modern environments. And so, I definitely second this advice. Any further reading links, anything?

I wanted to, because we got a little bit into how do we deal with like endpoints and user privacy and all that kind of components. I don't wanna be that guy, but I am gonna say that there's 2 technical things. I don't read a lot of tech blogs, but there are 2 things that I would love folks to read after this and can think about it. Number one, RFC 7258, Pervasive Monitoring is an Attack. I think you'll find that to be a fascinating read if you haven't already read that, and it does relate a lot to how we do observability and monitoring. Secondly, RFC 8890, The Internet is for End Users. It's a very philosophical approach from the IETF, right? But I definitely think it's a really interesting one, and I think it's something that we could be thinking about as security practitioners in terms of, at one point, the internet was meant for this class of people. Today, it's a different class of people, and that totally changes how we do monitoring. So there's that. And then finally, on the last reading recommendation is my son and I've been reading Ray Bradbury Martian Chronicles, which is a fantastic read. It's basically a series of short stories. Essentially, it's a really prescient view into attempting to settle on other planets. And I thought it was pretty close to the cloud, right? Because people are trying to go move from one environment to the other one, to a newer one, thinking that it's going to be better. but they bring their same old habits with them, the same old ideas, and the same problems. And as you see, space exploration is exciting, but it can be also very deadly.

So anyway, this is actually super valuable, and I think Tim would have loved it, would have loved to hear it, because like reading Ray Bradbury, flying to Mars, is kind of like cloud migration for a lot of people. So this is— it's now stuck in my head and I just can't get it out. And I think like a lot of the mistakes I've seen people do with cloud migration would be explained if they were traveling to Mars. Exactly. So that's Thank you very much, Jeff. That's hugely appreciated. Thanks for your time.

Thank you. I appreciate it.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.