Next 2022 Log4j Reflections, Software Dependencies and Open Source Security

Security.

Hi there. Welcome to a special episode of the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. This is a special episode for Google's Cloud Next conference. You can go see more at cloud.withgoogle.com/next. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website. This is going to sound familiar. cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday or on special days like this episode, please do subscribe to our show. You can follow the show and argue with your hosts on Twitter as well.

twitter.com/cloudwithgoogle.

Anton, we're learning a ton today about the software supply chain and why Log4j was so hard to fix.

Yes. Moreover, as appropriate for a security show, we're going to have a couple of really scary moments, like, is another Log4j coming? And will it be here by Christmas? Okay, for that one, we don't know the answer, but—

No, we don't know when it will be here, but it's a little bit like, when does the latest cryptocurrency rug pull happen? You can't predict it, but you know it's coming. I think what's fun about this episode is we have one of those classic heads, the attackers win, tails, the defenders lose moments when talking about different options when it comes to the pinning of software dependencies. And so there's a lot of great content, and I think without further ado, let's dive in and welcome today's guest. Listeners, I'm delighted to introduce today's guest. Today we are joined by Dr. Nicky Ringland, a product manager for Open Source Insights here at Google. Nicky, I'm thrilled to have you on the show. I know scheduling this one was a bit of an adventure with one of the podcast hosts in Europe, another in America, and you, of course, in Australia. Today we're talking about open source software. Can we rely on our dependencies? Are dependencies dependable? And Log4j. There's a lot to cover here. So I guess let's start at the top. Let's talk about open source software. People trust it, right? It's trustworthy.

Is it?

Absolutely. I mean, it's trustworthy in that people use it. So I guess the question answered itself, but assessing how trustworthy is a complex task. There's a modern concept of eyes on the code to assess finding vulnerabilities and then hopefully fixing them. There's a lot of questions about whether that's enough to rely on. Whether that's true or not, people do rely on it. Modern software is more than just lines of code that I've written myself and checked into a repository. Like, to build any program, essentially, you have to install packages from other developers, and these external dependencies are critical components of today's software environment, whether we're talking about like the tooling that's created to install dependencies and update them as well, or the dependencies themselves. I mean, it's trustworthy in that we all trust it. And this has created an amazingly vibrant, fast-paced ecosystem that is just growing at a phenomenal rate. But it also, yeah, there are challenges, let's call them challenges, that we need to address as well.

Uh-huh.

So I guess there's something interesting that you did there, which is you answered my question of saying, is it trustworthy, by saying, yes, people rely on it. Is there a difference between something being trusted and something being trustworthy?

Yes, definitely. And a lot of that comes down to, I guess, personal or organizational risk that you want to take on. So let's step back for a second. We're talking about open source dependencies. So I'm trying to write a program and I pull in some dependencies to get the code working faster, to make it shinier, whichever. But I haven't written that code, someone else has. And I inherently, by agreeing to use this code, form a trust relationship with them, someone I've never met, tens of thousands of people I've never met. And I am trusting them to not be trying to steal my credentials or my precious Bitcoin. There are some terrifying examples where that trust relationship has broken down. But the real question here is, how do we assess the risk of open source dependencies? And obviously, people across the ecosystem have assessed this risk and said it is a worthwhile risk, but a worthwhile risk to make hopefully informed decisions in.

That makes a lot of sense.

So this makes a lot of sense, yet we live in the world where disasters like Log4j happen, right? So it's almost like I want to use your sort of not overly optimistic but positive take on trust in open-source software and correlate it with a recent disaster such as Log4j and possibly other disasters, right? Like before that, we had OpenSSL disasters a good number of years ago, and there were other big sort of explosions, to use a possibly not very safe metaphor, with open source. So, Log4j though seemed to be As bad as it gets. Like, a bigger— as bad as they— I don't want to say as bad as we had so far, because obviously somebody would say, hold my beer, right?

Yeah, then don't tempt fate, please.

Yeah, it's like The Simpsons meme. This is the worst security breach of your life so far.

So far.

So why was it so bad?

Why was Log4j such a big deal at an ecosystem level? Let me ask you a serious question and avoid this kind of doom mongering to the extent possible.

So there are a few things going on here. First of all, remote code execution, which is the kind of vulnerability that happened in Log4j, they're a big deal, right? This is something that is as severe as it gets pretty much in terms of vulnerabilities and advisories. So it's something that if you were affected, you wanted to fix it immediately because people could run code on your machines. That's generally a bad idea. Okay, so there's the severity of the problem. Let's park that for a second. Another problem was that it was a zero-day exploit. So that means that there was no coordinated disclosure, coordinated vulnerability disclosure. So basically everyone found out because the attack was happening. This is generally a really bad way to find out that you're vulnerable to something. In an ideal solution, someone, maybe a security researcher, a white hat hacker, someone would find a vulnerability and say, I'm gonna do the right thing and let these people know so that they can fix it. And they would reach out to the maintainers privately and say, hey, there's this problem. Maybe they would do it through a bug bounty. There are lots of ways to make this moderately lucrative to quite lucrative depending. And that would allow the maintainers time to fix and mitigate the issue before— Everybody else found out about it. That did not happen here. So everybody found out about it and everyone was scrambling to fix it.

There is scrambling, frankly, right? Well, yes, some of them are still scrambling.

Sorry, the scrambling has slowed, but that's probably a reflection on which parts of who's using the software and where they're using it and how they're using it. But there's another really big factor for Log4j that caused it to be such a big deal. And that is just how much of the ecosystem used Log4j. So 8% of the Maven ecosystem depended on Log4j.

What is the Maven ecosystem?

So for Java, if you are going to use packages, you download them, you get them primarily from Maven or a few other repositories. But this is a proxy for saying about 8% of all of the Java—

Of all software, basically all software In that land, in that land, it's basically, it's like you wake up in the morning and you're like, by the way, 8% of your software is bad.

Yeah. Yep. And further complicating it, right, when you're choosing dependencies, you're not just choosing a bit of code to use. You're making this trust relationship on this dependency, which it itself has a trust relationship on dependencies, and they have dependencies, and they have dependencies. And it sounds like a nice clear tree. but it's not. It's a terrible cyclic graph and it's just horrid.

Ooh.

And you can only affect changes in your direct dependencies. So if you are using in this graph, and we'll call it a tree for want of explanation, if it's down a little bit, you have to hope that everyone else fixes it so that you can pull in those fixed remediated versions. And we found that because of Log4j being a logging tool. It was used everywhere. And if you've got Log4j 8 levels deep, you have to wait for 8 people to independently go in and manually fix this issue. So that's probably the, the last big thing of why Log4j was such an event is that because of how Java deals with dependencies, it wasn't a fix that could fix itself. So when we're choosing dependencies and when we're specifying which versions of dependencies to use, different language ecosystems have different conventions. And for Java, there's one other factor, which is that the Java ecosystem, the way that dependencies are specified, means that this didn't end up being a self-fixing problem. So let's go back here. What I mean here is how specific versions of packages are chosen depends on the requirements that are put on that package. And this changes based on different language ecosystems. Some language ecosystems like npm tend to preference the most recently released version of a package. And other ones like Go tend to preference the older and more stable ones. Now, in all of these ecosystems, there are ways to override the defaults, but generally language conventions are pretty ubiquitous. So in Java here with Maven, what is predominantly used is this thing called soft version requirements. And these default to something that's a bit more like Go. If you want, it will not automatically pick up the most recently released version of a package. So what this means is that every single person who was trying to fix the issue, who was trying to mitigate this issue, had to go in and manually update, manually bump their dependency versions, which if we're talking about 8% of the ecosystem is a lot of manual work.

So to make sure I understood this, there's really 4 reasons this was so bad. The vulnerability itself was quite severe. It was remote code execution. It was widely used. It was deeply depended on, and the language made it harder than other languages to deal with. Yeah.

Also easy to exploit. Easy to exploit. You don't need to craft complicated— Oh, yeah.

Sorry. The exploitation was trivial. And there's another reason, which was there was no coordinated disclosure on this one.

Yes.

So this was really kind of a perfect storm.

Yeah. Oh, and also more of a social issue.

Oh, and it gets worse.

But wait, there is more.

It was early mid-December. So—

It was like a week before Christmas.

Yeah. Everyone was not really ready for this. We were thinking about going on holidays.

It came out on like a Thursday night, right? Or a Friday night?

Yeah.

Yeah.

And then suddenly a lot of plans were changed and people were scrambling.

Yeah, I texted my TL on a Saturday morning and said, hey, Chris, this is pretty bad. I think you need to get online.

Uh, yeah. Well, anyway, it happened.

It happened.

I would love to say it's the worst thing that will happen, but it absolutely isn't. But what we can learn from it is— I mean, I don't know, you could learn a lot of things from it. You could learn that vulnerabilities are inevitable. You could Learn that actually maintainers respond ridiculously quickly, even when it involves manual effort. So the response time for this was phenomenal.

That's good news, right?

It is fantastic news.

Yeah.

Everyone.

Are we allowed this on a security podcast?

It's encouraged.

Oh, okay, good. It's a Google security podcast, so we are allowed. Yes. Perfect. Thank you.

I mean, I think everyone who was working on the Log4j response. definitely deserves an extra chocolate basket and, I don't know, an hour of puppy time.

Hmm.

Just maybe monthly as a reminder of their hard work and as a bonus to be ready for the next thing. But one of the other things that I and my team were really interested in is more around what the impact of Log4j being in Java was. And we did a couple of thought experiments. About, as I mentioned earlier, other language ecosystems have different dependency policies in terms of what gets pulled in automatically.

Yeah.

You mentioned this. Are other languages more secure? Is there a better way? Is there a worse way? It sounds like trade-off ways.

Yeah. That's a really hard question to answer.

But surely it can be worse than C, right? Like everybody, even somewhat sane, would hypothesize that it should be worse than C, right? C++.

I thought the point of Java was that it was safer. It's memory safer.

Yeah, wait, wait, wait. But I'm pretty sure Nikki has the answer.

Yeah, Nikki, please.

Nikki has a better answer than me. Oh, don't look at me for the answer here. This is way too hard. I have lots of answers, just not the answer. So some languages are absolutely more secure than others, and some have much better security policies than others, and some are just a mess.

Perl.

But it turns out that you may not have control of what language ecosystem you want to write in. I could say that moving away from C to something like Rust or Go would be great for your security team, but can you actually convince your TL that you should just port your entire codebase? Is that more of a security risk? Because what bugs will you introduce in doing so?

Yes. It reminds me of a good old security admin— system administration wisdom, right? I'd rather take a great admin with Windows than a bad admin with Linux. You can argue that one platform is more secure, or at least has been more secure in the past, but if you don't know it, then it's less secure. So, I'd rather take a competent Java programmer over a newbie, I don't know, programmer of C++.

That's absolutely— I absolutely agree with that. And I agree with that not just in terms of work, but also in terms of dependencies, right? So, when we're making— this is the thing that I would really like listeners to take away, that when we're making decisions about dependencies, they should be informed decisions. When we make decisions about sysadmins, they should be informed decisions. In terms of this thought experiment though, of looking at what the impact of these language conventions are on vulnerability issues like this, we actually didn't have to wait very long for our thought experiments to play out in real time because there was another security incident that happened in npm only shortly after. Mm-hmm.

Wait a second, which one is that?

This was the colors issue.

What language? Which programming language fell into that?

This is in JavaScript.

JavaScript?

In JavaScript.

As opposed to Java?

Yes, yes, JavaScript. So what happened here as this counterstory to Log4j is that a maintainer deliberately pushed breaking versions of a package. And this resulted in a lot of things breaking. A Colors maintainer actually pushed breaking versions out into the npm ecosystem, and this was picked up by a lot of users of the package. So npm preferences the most recently released version.

Hmm.

So what actually happened was they released 3 different versions. The first version was marked as an alpha version— sorry, a beta version, and it was mostly ignored. But then another 2 versions came out subsequently and they were picked up by about half of the packages that depended on it, which was around 50,000 dependencies. So we're talking about big impact here as well. In the essentially 72 hours that these 100,000 packages that depend on pkg were potentially affected, so about half of them were affected. So, I mean, that like— Happened like that. because anyone who built during that time took in the new version, suddenly were vulnerable. This is bad. But what can we do there? So this is the case where the— let's roll this back a little bit. If Java had a convention where it preferenced the most recently released version of a dependency, then Log4j— Oh, I'm gonna go out on a limb here and say would already be fixed. Everyone would've fixed it already because it would have been much easier and required much less manual effort. This is a counterexample where requiring less manual effort actually pulled in the vulnerability or the broken package in this case and made 50,000 people affected that otherwise wouldn't be. But the cool thing here is that the maintainers of some of these really popular packages that used colors noticed the issue. They pinned to an old version And the actions of about 40 maintainers fixed this for 25,000 packages, for about half of the packages that were affected.

Wow.

Within 48 hours.

But this is a modern stack and not Java, right? This would happen— it's very clear to me this would happen in a modern stack, but it won't happen in the 1980s stack like Java.

Java's not from the '80s.

Well, early '90s. I'm sorry.

Yeah, exactly.

But wait, Nicky, this kind of sounds like a heads attackers win, tails defenders lose kind of situation. You either lose because people get stuck with old packages and they have to rebuild, or you lose because people automatically pull in the backdoored code. What gives?

Pretty much. Yeah. Yeah.

There's no good answer?

There's no good answer. I mean, perhaps the right answer is to just give up on software. Maybe this was all a terrible mistake.

I agree with that. Networking computers was a bad idea.

Okay, fine. Finally, we are getting the doomsday back. I'm happy now because we are coming back to a doomsday theme. Otherwise, we may lose our security professional cards or whatever. So to really ramp up the temperature a little bit, I know it's kind of scary, but is another Log4j essentially inevitable? Do we have— well, we do have more widely used, hard to patch, many layers of dependencies, libraries hidden all over software. And I don't just mean SSL, by the way. I mean, other stuff, right? So what can we do? What can cloud customers, what can other organizations do to kind of minimize this risk? Assuming the next Christmas we're going to have— sorry, I'm not trying to predict it, but like some Christmas, not next Christmas, we're going to have another Log4j. What can we do to prepare?

Look, that's not the holiday gift I want, thanks. So what can we do to minimize our risk? I mentioned earlier that these Dependencies really are trust relationships. So an obvious way to minimize your risk is to minimize your dependencies. That sounds a lot easier said than done. But maybe just taking a bit of time to look at your dependency graph and think about whether all of those dependencies are actually required. So there's a Go proverb that a little copying is better than a little dependency. This is obviously moderately contentious.

Yes. I was about to say, wait a second, what?

Like many things in Golang.

But I think it also has merit, right? So think about having to make conscious decisions about the dependencies that you're taking on and whether the risks really do outweigh the benefits. Even if the answer is always yes, At least it's something that you've thought through rather than just taking on packages that are ubiquitous and potentially incredibly problematic.

I think that's really interesting, actually. To borrow a proverb from another community, this is like risk-aware community programming.

Exactly.

We're being very thoughtful about what we're doing. We're not saying there's absolute right and wrong and there's hard lines, but you should be thoughtful about it.

Yeah. I wish I could say, look, just install this one additional package and then everything will be fine forever.

Including for Java.

But it turns out that that will not work well. There are other ways you can minimize risk. You can do dependency audits. You could have weekly dependency triage. You may drive everyone a little bit wild with this focus on dependencies rather than making progress in the things that people want to do. It's all a balance, right? People rely on open source because of the power of open source and the velocity and all of the shiny, awesome bits of open source. I don't want to stop that. I want to encourage that. Part of what Open Source Insights and deps.dev is all about is providing open source maintainers and open source users tools to assess dependencies and to improve the security and overall health of the resources that they use. And we've got really shiny graphs as well. I don't know if I can give a shout out, but like really shiny graphs.

I like shiny graphs.

Who doesn't love shiny graphs? And of course, there's one other thing that you could do is look at perhaps I don't want to say mitigating some of the risk associated, but looking at products like Google Cloud's Assured Open Source Software service to make someone else worry about the risk on your behalf. I'm not saying this absolves you of all risk or of all interest, but that's certainly something that organizations could look into.

And what is the Assured Open Source Software service? What's the 30-second elevator pitch on that? Because to me, that's a really cool thing.

Yeah. So Assured OSS is a service that's brand new, provided by Google Cloud to help mitigate a bunch of the risk of dependencies. So it lets Google take on a lot of the maintenance and like a health and assessment burden to help make your life a little easier. But you should still check out the Shiny graphs on deps.dev. Either way, you can check out More around Assured OSS at the link provided.

And we'll put the link in the show resources, listeners. So with that, let me ask my traditional closing questions then, Nikki. Recommended reading beyond the links that we're going to provide in the show resources, listeners, and one tip to improve people's software dependency security.

Oh, okay. So recommended reading. If you enjoyed this chat today, you would probably enjoy some of the things on our blog where my team uncovers some curiosities of the open source ecosystem. We just published one the other day on mean time to remediation and whether this is a good way of assessing the risk of a dependency. blog.deps.dev is where to go for that. Also check out deps.dev and— have I plugged the shiny graphs enough? They're really shiny.

I'm starting to wonder if they're shiny at all, frankly.

Okay. So let me take a step back. This is one of the ways Okay. Maybe this is the tip, right? If you suddenly find yourself impacted by the next Log4j because you're using open source dependencies and it's somewhere in your dependencies, 8 levels down and you don't know, and it's— what do you do? The interactive graphs that we have on deps.dev, you can click through on the dependency, look at its dependence, and search for all of the dependencies that it has in the graph that are impacted by advisories or impacted by Log4j. And that means that what you can do is immediately assess whether this is even achievable for you to remediate right now, or whether it is just a spiral of doom and despair and you should go for a walk and come back next week and try again later once some other versions between you and it have been fixed and patched and pushed.

Amazing. Well, Nikki, thank you so much for joining us today. Listeners, I hope you found this hopefully not too frightening.

I hope it wasn't frightening. I still think open source is fantastic and you should use dependencies and you should know what you use.

Absolutely.

Maybe that's the takeaway.

I think that is the takeaway. You have to be risk aware and think things through.

And go for it.

And go for it.

Make a decision and build something awesome.

A note of optimism.

Thank you.

Thank you for joining us.

Thank you so much for having me. And thanks, everyone, for listening.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.