EP54 Container Security: The Past or The Future?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and subscribe to the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. And with that, I am delighted to introduce our guest today, a title so impressive Anton could not bring himself to announce it. I am pleased to introduce Anna Belik, the Director of Thought Leadership at Sysdig. Anna, thank you so much for joining us today. It's a real pleasure to have you on the show. I want to kick things off with kind of a framing question as we think about container security. I think of it as, you know, infrastructure security, build security, and runtime security. A, do you like this framing? And B, if you like it, what do you think is in there the easiest to get right, the hardest to get right? Where are you seeing investment for users to try to help them get it right? Talk to us about the general landscape on container security.

First of all, thank you for having me. This is a huge honor. I am a total fan. I think I've listened to maybe all of them or very close to all of them.

Aw, shucks.

And you guys rock. So, very happy to be here. It's very hard to answer your question, actually. I want to say I'm going to nitpick you from the beginning and be like, there's actually application security too.

Hello.

Okay. But I also don't want to talk too much about it because application security is a whole other can of worms. But it's a really important element. The reason we actually care about containers for the most part is because it eases life for developers, right? That's the driver. And that ends up really important in how we then manage them, secure them, and do everything else with them in the end, right? But to your point about the 3 things you mentioned, I do actually agree that those are all key. In classic nature, your question's really hard. I think the hardest one is runtime. There's a couple of reasons for that.

Runtime.

The easiest reason, I think, is one that's near and dear to both of your hearts, and it's that detection and response is just hard. When you try to do detection and response on containers, it's still hard for all the reasons it was always hard. And now it's hard for some new reasons, like they're ephemeral and they do silly things that look wrong when they aren't, and so on and so forth. I think that the most important thing, actually, or maybe the thing that we're spending the most time and energy on right now, is more on the infrastructure side.

Mm-hmm.

Because before you can really worry about runtime, you have to get some basics right. And this almost goes back to, like, a security hygiene story of, look, you have to, like, not open silly ports. You have to scan for vulnerabilities. You have to, like, think about your configuration being secure. So, all of those things are required before you can get fancy with runtime stuff.

That makes a lot of sense. We kind of have a missing piece from that set of security, which is, of course, And then of course, data forensics and incident response. I just assume people should earmark 10% of their budget for that and call it a day. So that's an interesting answer, and I think a good one that really starts at the start of the list, which is getting your hygiene right.

Perfect. When I think of hygiene, I think of infrastructure first. Maybe I'm just not very AppSec-minded or something. When we think about Sysdig and similar technologies helping users get their infrastructure security right, but what do they get wrong? Because again, reading your report, which we'll mention more throughout the episode, it does seem like users and container users would sometimes get infrastructure wrong. So before we get to all the fancy stuff like runtime and build security, they actually miss the infrastructure angle. So what do they get wrong most of the time and how do you help them?

This is, I think, nuanced actually, because what you mean by infrastructure becomes important, right? Like, is a container a piece of infrastructure or is it a piece of application? It's like a real existential crisis question that I actually struggle with. I think that the thing that they get wrong is kind of higher level than any of that. And it's the pattern that they're trying to impose upon whatever it is they're doing, whether that's an application they're building or an application they're migrating or whatever, right? The beauty of containers and Kubernetes and cloud native and all these other things is that you're trying to operate in a different way, right? Like the cattle, not pets thing, where you can just throw away images that are broken instead of trying to fix them. You're moving a lot faster. You have a lot more redundancy built in by default. You're able to scale. So, all these things give you a lot of power. What often happens, which I think is the biggest mistake, is our users don't actually take advantage of these things. They take their legacy habits and their legacy ways of doing things, and though they've moved their infrastructure and their application to the cloud, they haven't actually applied to it any kind of cloud-native philosophy or approach. And that causes them to do things like not patch often enough, or rather, I shouldn't even say patch because it's not patching, it's just immutable. You destroy it.

So, this problem of users Coming to containers thinking containers will fix their things. That's like going to Hawaii with your partner and things aren't magically better there. Containers as Hawaii. You say in your report, I think I can quote it as saying, 3/4 of running containers have at least one high or critical vulnerability. To me, this is like the classic example of people adopting tooling thinking it'll fix their problem and not having it fix their problem. So why aren't we running latest? Why aren't we redeploying? What's keeping people from having the outcome they want?

I think you're absolutely right about the cause, but I think the solution is just not simple because it's never simple, right?

Hmm.

First of all, there are things that should be easier, right? It's quicker to get an answer to whether or not this thing is vulnerable. It's quicker to throw it away and deploy another thing that's not vulnerable. But whether or not you can actually remove a vulnerability is still not that trivial of a question, right? There's still dependencies, right? So you have components, there are other components dependent on those components. This results in a horrible supply chain problem that causes Anton existential dread. At the end of the day, you can't actually make the decision about whether or not any given thing can be replaced trivially without affecting lots of other things. You still end up with these risk decisions and these workflow problems that cause people to have to kick this can down the road. And at the end of the day, they're incentivized to release features faster. They're trying to build software, they're trying to deliver business value. And so the security thing ends Every moment that it slows them down, it just gets ignored or dodged.

That sounds like a lot of pre-cloud thinking, right? Like, as you mentioned in the question, it sounds like a lot of the approaches and practices and habits from before cloud come in, and people sort of follow them. So, for example, you just said it's easy to scan, find the vuln, but then you also said right away it's easy to destroy it and put a new one that's not vulnerable. People do the first part, but not the second part. Here, we're not asking them to patch. We are asking them to take a flamethrower and burn the old container to the ground, get a new one. That is supposed to be an easier task, yet they don't do it. Am I right or am I still too—

You're right. But consider the implication, right? So if the vuln is in some, let's say it's a load balancer, it's like a single thing. It does one thing. You have a new version of this load balancer, you just destroy it, you deploy a new one, and it's going to go ahead and route your traffic, right? That's pretty simple. Now consider if the vuln is in a database. What if it's Cassandra? And so now you have to patch this vuln in Cassandra. Okay, I'm not going to patch it. I'm just going to toss it in the trash and deploy a new Cassandra. But all of the stuff I built that connects to that thing, that relies on that thing, may or may not be able to handle the new version trivially, depending on how it's architected and how it's built, right?

Mm-hmm.

That's a risk, right? That's a business risk to just deploy that thing. So, you have to consider the implications of that. Even if the fix itself in theory is simple, the outcome may be painful. Yeah.

And you can't have a case where the medicine is worse than the disease here.

Exactly.

Sometimes you just have to choose to live with something not being quite right.

Right. But this whole cloud-native stuff is fake because this is the same problem I would have if I'm patching my Oracle and the patched version 8.5.7-23_45 does not work with an app.

Anton.

Like, this is the problem I would have in 2007.

Anton, you can't say the cloud is fake. You work at Google Cloud.

No, no, no.

I didn't say cloud is fake. I said cloud native is fake.

Ah.

Oh, man.

Because I said that the problems you're describing are the same problem IT would have with patching a database in 2007. Where's the magic?

Nothing ever changes. I mean, there's some magic. So a lot of things do go faster, for example, right?

Yes.

The amount of time it takes between when something is disclosed or when you discover something is wrong and when you can actually fix it is much smaller now, right? You can have a release the next day that fixes everything, potentially. So I do think that it's a little unfair to say nothing's changed because in 2007, this might have taken months or years, and now it might take weeks or months, which is like a pretty big improvement.

Okay, I'll go with that. I do want to switch to the positive theme. And one other thing from the report that they picked is that while 75% of containers, as we just said, contain patchable vulnerabilities, some customers, the most mature and meticulous customers, as you say, reduce this metric to below 5%. 5%, 75% sounds like a big difference. So as a longtime maturity, kind of operational maturity nut, it sounds like there's a bit of a chasm, possibly a big chasm between the average 75% problematic and best 5% problematic. How are they doing it? And what's the magic behind them being so much better? Are they the true cloud native? Do we now have the cloud native and like the real cloud native? Or is there something else going on?

We've got some Scotsmen about to join the call, I think.

I do actually think there's something to the cloud-native in quotation marks versus real cloud-native. My personal opinion is that I do attribute a lot of that success to folks actually adopting some of these patterns. That includes not just the immutability thing of throw it in the dumpster, deploy the new one, but also the shift-left buzzword, which actually just means to find these problems sooner. The sooner you can discover there's an issue, the easier it is to actually alleviate that dependency suffering for the developer who is responsible for making sure that the issue isn't there. From our perspective, the folks that use at least our tool, they scan sooner, they are more serious about making sure that they get visibility into what all these workloads are doing and how they rely upon each other. And they generally just have, they're doing a better job of that hygiene stuff in a cloud-native way that is specifically How you should treat a container, as opposed to trying to compromise between this legacy approach and this new approach.

That makes a lot of sense. On the topic of shifting left, and on the topic of build, we saw in your report that 52% of images are scanned at runtime, and 42% are initially scanned in the CI/CD pipeline. That seems to me backwards. Shouldn't most people be pushing their scanning into the build pipeline rather than the runtime? What's up with that?

Again, this is where I would say that the folks that are higher maturity are definitely pushing it into their build pipeline. In fact, they're actually doing even more advanced things where they scan their artifacts. They'll scan infrastructure as code, YAML, before they ever build anything. So, they can see misconfigurations before they've ever bothered to build a thing. The issue is, I mean, first of all, again, there are folks who are just picking this up. It doesn't take 5 minutes to build a CI/CD pipeline. So, if you've never done that before and you're just starting to adopt some of these practices, it could take a really long time to build out that.

How long?

Years, potentially, depending on where you started, right? It's not just about buying a trinket and plugging it in, right? There's a lot of process and culture that has to shift, right?

But years, you make it sound like these people are melting down sand and building their chips from scratch by hand.

I mean, it's pretty close in some sense.

So really, you think this is like a maturity thing and the difficulty of getting CI/CD stood up and doing it right and all of that?

It is a maturity thing, and a lot of it is also just skill set and process and training where people just need to get used to the way that things go. Sure. It takes some tuning, right? So, when we say scanning CI/CD, that means you have to make a decision. For example, at which point do I fail a build? At which point do I alert on a build? And at which point do I just ignore it because it's never going to get fixed because it's so low or whatever, right?

Mm-hmm.

Those kinds of gates take tuning because you put something in, it does or doesn't work, the developers hate you, then you go back and forth and you negotiate. So, this all takes time to get right. The other element here that is interesting is that part of those images that are scanned at runtime for the first time, actually a non-trivial part, are things that aren't actually built by the developers, right?

Yeah.

So again, they may be load balancers, Kubernetes components, or other kind of what we used to call middleware back in the day, that you're pulling down from a repository, or you're using it as a vendor-consumed piece of software that you trust, right? So you assume it's not vulnerable. That may or may not be a safe assumption. So again, the best practice is to scan all that stuff in CI/CD too, and have it be part of your testing process to check all those pieces. But a lot of people haven't done that yet because, you know, maybe it's on their roadmap. Maybe it's just not something they think is as important as scanning the things that they do build. There's a different risk profile associated with not scanning those things because you should be able to trust your vendors to some extent, right? Rather than not scanning the stuff that you actually put together yourself.

So people aren't scanning their shrink-wrapped code. That makes a lot of sense. Or at least they're not scanning their CI/CD.

Okay.

Yeah.

But that's disturbing also, right? As if there's not more disturbing stuff to find, because this is kind of something that I recall from my Gartner days where they security shifting to pre-run time. And frankly, I was nervous we're going to lose the runtime security because people would supposedly do all the security in the beginning. And then they would claim that they're all done. So they would not do runtime security. So that was my fear back in the day. It turns out the fear is not materialized, but something worse did. They're not doing enough security before deployment.

I think you were just ahead of your time, Anton, actually. I still share this fear, honestly, that people will think that they can check all the boxes because They have this beautiful testing pipeline that makes sure that everything's okay. And then they can throw that thing into production and sleep at night.

Yeah. That's my fear from 2018.

Yeah. Like being a paranoid security person like you, I'm over here like, that's a terrible idea. No, it's going to be awful. And that's part of what kind of we do actually. We try to provide that runtime context for folks, but at the same time, it's hard to really expect good runtime hygiene when you don't have good preventative controls yet. So again, I think you're just ahead of your time in that these things will happen, but they haven't happened yet.

This is almost starting to sound like in this call, we're backing into some kind of, dare I say, capability maturity model for container security.

You want to write something about that?

No, I don't. That's your job. You're the director of thought leadership here.

That's right. There's another element actually that is kind of interesting in terms of people and process, right? In that, in the old days of vulnerability management, we had this struggle of deciding who the owner of something was. When you push something through whatever scanning process to identify what's wrong with it, your next question is, who's going to fix it? So, you have application owners or system owners or what have you. And usually, they were in I/O and they hated you, and so that was painful, right?

Yep.

Now, there's actually a different but maybe more complex issue because you could have an image, like a container image, that has different layers that are owned by different people, right? Its base image that is an operating system or whatever enabling layers might actually be a DevOps or ops Ownership element. And the things that are sitting on top of it that the developers created or the dependencies they pulled in, like third-party libraries, are the developer's job. Tooling that exists today isn't super great at being able to tell which is which and whom to ship that ticket to. And so we still have that issue of whose job is this? And so who's going to fix it? Nevermind, like when they're going to fix it.

Hmm.

Okay. No, basically to summarize it in a somewhat cynical manner. Security used to mostly fight ops and occasionally rarely fight developers. But today it looks like security will just fight DevOps, which is roughly the same people they used to fight, just on a different group name organization.

That's exciting. They've been rebranded. Yeah.

Okay. So switching gears again. So one thing that I also caught up on the report, and again, I wasn't trying to read the report as it's a story about how IT and security were done in the past, but there was a line about scanning third-party containers and runtime being popular because it reminded people of a simpler legacy approach to VM when they used to always scan servers as they're running. What it sounds to me is that it's not very cloud native. To me, that sounds like a classic premise on-premise security where you build something, you deploy it, you push it into prod, and then you get a scanner, you unpack your little vulnerability tool, and then scan it. And it's like, oh, there are bugs, go fix them. Why is it still there with us? Why is this practice still around? Why didn't it die when people went cloud native?

I think the simplest answer is that people stick to what they know, right? And you have to start somewhere. We try not to judge them. We just kind of want to secure them. So there's an element of meeting them where they're at. But I do think that that's the tendency, right? If you can't get your developers to adopt some of these best practices of shifting left, if you can't afford to build this beautiful pipeline, you have to do something. And so you do. You start at the end, right? You start it all the way to the right. So, you can at least fail the builds that are the most broken or what have you, have some standards, and then you try to go to the left.

Okay.

Back to your other point about the terror of runtime security is that you can't assume that because you've scanned it on the left, it's okay forever, right? So, you still have to scan it in runtime, probably scan it forever in runtime continuously. It's just that that shouldn't be the first time you scan it.

Yes. I think it does make sense. And I think that it's nice that we know of at least one problem that would materialize in 2, 3 years at many companies is that when they would shift too much left, and they would not do enough runtime, then we'll be here with like, hey, hey, we knew it back in 2022. Okay. So, I want to ask another question from another thing that kind of like blew my mind a little bit from the report. There was a metric about 72% of containers having shells in them. And a quote from our podcast notes is that, to me, it sounds like 62% of zoos have a dragon in them. It's like really bizarre, surreal. Why are people SSHing into freaking containers? What's going on? This is not supposed to be like this. Can you care to comment on this one?

It's 62%, but yes. I think it's the same reason, right? First of all, again, I have to be a little bit empathetic, I think, to our poor users in that when you try to modernize a workload, which is what's happening, some of this stuff is definitely greenfield where people are building brand new, beautiful, shiny things. But a lot of businesses just have legacy apps that are running that are business critical, and they would like to refactor them to be newer and shinier. And that is not a trivial process, right? That's a very long road. They may well be taking something that is a monolith that has been running in a virtual machine for 30 years and moving it to cloud in the hopes that that's going to save them money or make it more likely that they're actually going to do something novel with this thing, or just because they were told to, I don't know. And then they're going to hopefully over time start parting it out, right? So piece by piece, they're going to dismantle it and build it into something. More microservice-oriented, perhaps, right? Maybe that's the dream. And so, if they're still in those early stages of, this is just a virtual machine to me, they're going to do those things. They're going to see some weird behavior. They're going to SSH into it to, like, yum stuff, what have you.

Oh, God.

And yeah, to me, that's utterly shocking. I'm over here like, it's a container. What are you doing? But at the same time, old habits die hard. And if you don't have any other way, or if you haven't actually designed it to be Kubernetes orchestrated and load balanced so that if you kill it, it'll fail over to the next one, then you don't have a choice.

If you don't own the build process for a container, it makes sense. You have to SSH to it and apt-get update. Those are your options in life. And if somebody doesn't have one, they're going to take the other, right? In some ways, Anton, I think we ought to be applauding these users that do go ahead and try to update their stuff, even if it's not, you know, a fully container optimized operating model.

Okay.

Wow.

You found the nice optimistic bit in that.

Yeah. I think that's my job.

I have an optimistic take on it too, actually. Going back to what we're talking about here, right? This is like the Sysdig. Usage report and security report. So, these people have engaged with a security vendor that brands itself as cloud native, right? Which means they either believe themselves to be cloud native or believe themselves to be on the road to cloud native. I feel like the fact that we are seeing these scary numbers actually indicates that the adoption of cloud and containers is broadening to folks who are not as mature, right? So, they don't quite know how to get it right, but they're trying to do it. And that actually gives me a lot of hope because, you know, I've been covering this stuff since like 2015 and I've always been like, wow, this is so cool. Why isn't everyone doing it? And now it feels like maybe everyone's doing it, right? Which is great. Like, that's awesome.

This conversation has made me think of the thing people used to say at conferences, oh, the cloud is just someone else's computer. Are containers just somebody else's computer? Like, is that what we're seeing here?

Oh, Tim, you know better. Come on, you're baking her.

Containers are your computer. Containers are a tiny piece of someone else's computer.

Yeah, that's what it kind of sounds like though.

It sounds like containers, for all their hype, have all the same problems as people's computers.

I actually would argue that the problem isn't that people don't love them enough, it's that they love them too much.

Oh, let them die.

Let them fall over. Like, why are you treating them like these virtual machine pets you used to have? Like, they're actually still your computer. Let them be someone else's computer.

I like that answer. So we're getting towards the end of the show, and we have our traditional closing question, which I think I get to ask both parts today. First, do you have one tip for people to improve their container security outcomes? And, you know, this is the part where I want you to do more than just say contact Sysdig. And then 2, do you have recommended reading? And go beyond just the report, which we will be linking in the show notes, listeners.

You should totally contact Sysdig. We'll help you out. To be honest, it kind of goes back to what I said earlier, right? I think if you are on this journey to cloud native, it's okay if you're not killing it from day one, because it's actually very complex and there are many, many things to learn. So don't hate yourself for the fact that you're not killing it from day one, but do try deliberately to actually adopt the cloud-native philosophy and the cloud-native patterns and not just the cloud-native tools, right? Because just doing the tech thing, as always, doesn't get you very far. You have to do the process thing.

That's a great answer. Ana, thank you so much for joining us. Thank you for all of the laughs, and thank you for letting me think of containers now as the tropical vacation for a struggling couple.

I didn't actually tell you what to read, but I can if you want.

Oh yeah. What would you like our guests to read, please?

I mean, read the report, obviously, which is awesome, and it's at satsync.com. And in addition, for folks who are just learning about Kubernetes, we have a pretty cool thing called the Cloud Native Learning Hub, which you can check out, and it will teach you basic things about cloud and Kubernetes security.

Amazing. Thank you so much.

And now we are at time. Thank you very much for listening, and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. Today, I will be particularly insistent, and I would say, please subscribe so that you don't miss episodes. I don't know. I should probably do it with my Russian accent. You can also follow us on Twitter at twitter.com/cloudtechpodcast. We are also on LinkedIn. We now have a whole separate page for the podcast. You can find your hosts on Twitter @Anton_Chuvaikin and @TimPeacock. Tweet at us, LinkedIn at us. I don't know what it means. Email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.