Sponsored: What npm 12 fixes… and what it doesn’t

Hey everyone, this is Casey Ellis for the Risky Business Podcast. We're here with Forossah Boukhadidje from Socket. Uh, we're talking today about some of the, uh, the changes that have happened in the npm ecosystem, um, where they've worked, where they haven't, and what Socket is seeing out there in the wild. Good to see you, Forossah.

Yeah, good to see you, Casey.

So yeah, let's, let's get started, man. What's, what's been going on?

Yeah. So npm version 12 just went GA earlier this month. And the big headline is that install scripts are now off by default. And so quick context for anyone who doesn't live in the JavaScript world. When you run npm install, a package can declare scripts that npm executes automatically as part of the install. They are called pre-install, install, and post-install. And they do exist for some legitimate reasons, mostly compiling native code. But the security of them is just absolutely horrible because the moment that you install a package, it can run arbitrary code as you on your laptop or your build runner. And this happens before a single line of your application has even loaded. You don't have to import the package before the attacker can run their code. And so as of version 12 of npm, they flipped 3 of the settings So now install scripts don't run unless you allow them. Get dependencies don't resolve unless you allow them. And finally, dependencies pulled from arbitrary remote URLs don't resolve unless you allow them.

Okay, so that all sounds like good news, but some of what you guys have been seeing is that there are some gaps in that strategy.

Yeah, I mean, I think, you know, attackers are really quick to evolve. And so one of the things we saw right away was folks started relying on this implicit behavior where if a package ships with a file called binding.jip, then npm will just kick off a native compile and it'll allow the attacker to run code through there. It's been interesting to see how quickly folks have evolved and worked around these changes. While these changes are really good and I'm glad to see them, I think it's the right direction. By the way, I think it's frankly been a little late. These changes have been a little late in coming. Socket's been kind of flagging these risks, install scripts, Git dependencies, and remote URL dependencies in our product for 3 or 4 years and flagging those for all of our customers. I'm glad to at least see the default package manager finally taking that position. But I think the key thing to think about with this change is that all an attacker has to do now is move their payload from the install script into the package itself. And now the moment that the developer actually imports the package, the payload runs. And so the question I ask is kind of like, how often do developers install packages that they don't intend to later import? Typically, you install it because you want to use it, right? And so this really just kind of moves the timing of the payload just kind of a minute or 2 later, which is good. I definitely think it will help. But I think we've already seen a number of attacks that have made this shift already in real time in the middle of ongoing campaigns in order to basically work around these mitigations.

Yeah, let's talk about that a bit more because in the middle of all of this, and you've got Socket's position in the ecosystem, you've got obviously your background as the JavaScript supply chain guru in that sense. So you've obviously had a lot of experience and a lot of visibility on how this has all evolved. over the years. But then on top of that, there's a lot of growth that Socket's experienced, and that gives you a lot of insight into kind of what's happening in the wild. We were talking a little bit before we started recording on, you know, you actually seeing some of the campaigns shift like midstream around some of these changes. Can you go into that a bit more?

Yeah, for sure. So let me, let me talk about one campaign. This one started early in July. It was a package called jscrambler, which is a commercial code protection product and they have an npm package that you can kind of wire into your build pipeline to use their product. And someone published a release of it with an install hook that never existed in any prior version. So Socket.IO, the thing we do, like our bread and butter is we look for this stuff. So we flagged it within 6 minutes of publication because we're doing this sort of broad ecosystem-wide monitoring and surveillance. So we can flag these things really quickly. But I think the— I guess the kind of issue with— let me walk through what the package actually did just to give people some idea of what the payload was. Yeah. So it was an info stealer. They had different binaries for each operating system. They hid it inside of a file. And when it ran, it kind of searched the system for crypto wallets, browser extensions like MetaMask, and so on. But they also kind of targeted cloud providers. So any access to the GCP metadata service to steal service account credentials, the ECS task metadata endpoint, Secrets Manager, all these kinds of things, as well as Discord, Slack, Telegram, every Chromium browser. And of course, the AI coding assistants are a really ripe source now. So they look through Claude Desktop, Cursor, WinSurf, Zed, all the different config files because a lot of folks put their API keys into those, for their MCP server configurations into those files. And so what we saw and kind of how the attacker adapted to your question was basically after about 3 hours of us flagging this attack, we saw the same actor publish additional releases as they kept trying to evade the blocks that were going out. And the payload was exactly the same, but the kind of way that they triggered the The delivery of the payload just moved from the preinstall script to what's called a self-executing function in JavaScript, which just means basically something that kind of runs right away as soon as that file is imported. You don't have to call any functions. You don't have to necessarily use the package by invoking methods in it or functions in it. You can just sort of import it and then boom, the code is kind of run as part of the setup of the import. And so it immediately moved around this npm v12 protection. It also defeated scanners that I'd say there's some lesser scanners out there that only inspect install hooks and they're pretty naive. And so those scanners then started reporting that the package was safe. I think it was interesting to see mid-campaign, within just a few hours, that type of evolution of their strategy.

Yeah, no, definitely. I mean, it's interesting to me, the game of kind of cat and mouse that goes on in ecosystems like this one. And it's not too dissimilar from what you see in other kind of versions of the same thing out there in the industry across the board. But some of the stuff that you guys have worked on to basically kind of bypass that problem and kind of approach this whole idea of package security from like a first principles standpoint, do you want to go into that a little bit?

Yeah, definitely. This is kind of a, I don't know, like a, I don't want to say pet peeve or just sort of a hobby horse of mine to talk about, which is just this idea that we can look, at the end of the day, our position is you really have to know what the code is doing. That's the real source of truth. That's the fundamental kind of trust question is should we be using this code because what is it going to actually do when we run it? What data is it going to access? What endpoints is it going to connect to? Maybe a way to make the point would be maybe tell a story of another attack that we tracked. Is it cool if I do that?

Yeah, totally.

Yeah, so there's this other package a couple of days actually from the other one, earlier this month in July, called AsyncAPI. This is a tool for event-driven APIs and ships as an npm package. By the way, I keep talking about npm just because it's the fun one where all the noise and fires are and we all love to pay attention and look and point at them and say, look at how bad npm is. But I just want to point out that we see this across all the ecosystems. We track Python, .NET, Java, Ruby, Rust. It's the same thing everywhere.

There's definitely— But as you say, npm is the currently topical cluster of trash fires that are drawing attention to the broader problem, right?

Absolutely. Yeah. I also want to draw attention to the Chrome extensions and to the VS Code extensions and to the cursor extensions, all the editors, different skills, anything that a developer is bringing in, that's all part of the supply chain. But anyway, I digress. Back to the story about AsyncAPI. Basically, this is a package that helps you use these event-driven APIs. They inserted the malicious code directly into legitimate source files inside the package. They put it into one of the utility files, into an error handler. And so they would run right away as soon as Node loaded the module. So again, bypassing the new protections in npm v12. But I think what was kind of interesting was, first of all, they spawned some hidden processes that used pretty much every decentralization technology that I've heard of. And I used to, by the way, I used to do, I used to develop a torrent app. So I know about a lot of the different dweb technologies. So this thing used like libp2p, the BitTorrent DHT, IPFS, Ethereum smart contracts, Nostr relays to do their C2. So it was really quite an impressive slew of basically every possible C2 channel. And it's kind of sad to me as a decentralized technology person to see kind of the killer app often being these C2 use cases. But I think the part that was actually most disturbing was that the attacker here didn't steal an npm token. They actually just got a commit into the repository and the project automatically built the project and then published it through GitHub Actions. So they have this kind of publish workflow set up that automatically triggers publishes when certain conditions are met, triggered through GitHub Actions. And it uses the OIDC flow that the entire industry has spent 2, 3 years telling all maintainers to move to so that we can do provenance and have integrity and do SLSA provenance attestations and all this basically fancy cryptography to help determine that a package hasn't been tampered with. And in this case, it didn't matter because the final artifact was malware and it had all the signatures and all the cryptography signed off saying, hey, this was built in GitHub Actions. It was built in a trusted environment. It was built by the correct GitHub Actions runner and nothing was tampered. And so it didn't really matter in this case because someone got to commit into the release branch and the project's own release pipeline just did the rest.

So it's just very well-dressed malware at that point.

Yeah. I mean, look, I don't want to be too harsh on it because I think that trusted publishing has a place, but I think it never answered the question about whether the source code was clean. the current supply chain wave we're dealing with is the question we want to answer. And so I think signatures all the way down, cool, great, cool to see that the commits are signed, but ultimately, you don't want to have a beautifully signed, fully attested, verifiably reproducible piece of malware, basically, right?

It's interesting. So, how do you— I mean, I guess there's sort of 2 kind of Angles on the problem here. One is this idea of these kind of cat and mouse shifts being useful, but not necessarily complete in terms of how they kind of address the overall ecosystem problem. And then this idea of things getting kind of dressed up even better and looking more credible, even though they're malicious as a result of that. How do you solve that problem, I guess?

Look, I think that I don't want the takeaway to be that provenance is completely theater because I think that would be the wrong lesson. But because I do think it worked in some ways. It did give us the workflow, the branch, the commit, and the runner where the package— basically where the attack took place. And so 6 months ago, I think that would have been a harder question to answer. How did this get published? It would have been like, who knows? Maybe the maintainer lost their token. We don't really have any other details. So it does give us a bit of a forensic trail and that is real. But I think it still doesn't tell you whether the artifact is safe. And so I think we just need, like everything in security, multiple pieces working together. And I think from our perspective, the Socket perspective, the really hard part is how do you take a massive pile of completely untrusted code, often in dynamic languages, which are notoriously hard to analyze, and figure out, do I let this into my application? Do I let this into my environment? Do I let the vibecoders in my company that are writing code, in some cases unknowingly, as Cowork kicks off some job and starts installing packages? How do we decide whether we want to allow those artifacts into the environment or not. So I think it needs to be this multi-level approach. And hopefully that doesn't sound like a cop-out. I just think every piece of the ecosystem plays its role and has to work together here to get us where we want to go.

Yeah, no, totally. To me, that doesn't sound like a cop-out at all. It sounds like a fairly familiar story across a whole range of different problems that we've tried to solve in cybersecurity over the years. Like, you've got these deterministic gates and things that you know that you can do that are going to improve kind of the situation from the outside in, but ultimately they don't necessarily address the first principle that you're trying to think through from a risk and a security standpoint. So to your point, like, approaching it as a pincer move does seem like the best thing to do. It seems to be the thing that we've done in a lot of areas.

Yeah, totally. And speaking of multiple levels, I think there's another thing I want the readers to— or the listeners to know, which is that there's another— there's kind of a change coming down the pipe that anyone using JavaScript should definitely be aware of, which is by the time people hear this, the first of 2 big npm token changes is kind of coming down the pike. So the first one, which is going to have already happened by the time people hear this, is that access tokens that bypass 2FA are going to lose a lot of powers. So you're not going to be able to use those tokens anymore to change maintainers, to edit org membership, and things like that. So it'll kind of completely close the account takeover surface. But publishing from those tokens is still going to work until around January 2027 when they're going to kind of shift it. That's the second stage to what's called a staged publish where you can use the API to stage a publish, but ultimately you're going to need a human with a 2FA approval to actually ship a package, which is another change I'm actually pretty excited about because I think that's going to probably meaningfully move the needle there.

The implications of that are interesting because you could kind of reasonably expect a last hurrah over those 6 months as well, right?

Definitely. I mean, well, not to mention all the stolen credentials that people have already collected for npm maintainers over the last number of months that I think are going to keep keep hitting. But I think the one thing I want to flag too is that this change about blocking install scripts, it's sort of nuanced in that the new default behavior is that when a package has an install script, npm just kind of ignores it. It's kind of a soft fail and just proceeds with the installation and it just reports in a console that an install script was skipped. But I think what's kind of nefarious about this is actually that oftentimes those scripts are legitimate. I mean, they are building your embedded databases, your image processing libraries, your fast password hashing libraries which are built in C and that do need to be compiled. And so what's going to end up happening here, my fear is, I guess, is that those are going to get skipped and you're going to get production failures, production errors. Those are going to be runtime errors when those calls happen and there's no compiled module for Node to call out to. And so what I think a lot of teams are probably going to do there is just blanket allow all install scripts to make the CI go green, right? And so I actually recommend folks enable an opt-in setting that turns the soft failure into a hard failure. So you have to make a kind of concrete decision for every one of those scripts at the time you bring in that package. And that's my recommendation for teams today for that.

Very cool. Very cool. Well, that's, I mean, that's a lot. There's a lot happening in this ecosystem. It's great to have you guys kind of wrapping your arms around the entire thing and helping us all stay secure. Everyone, this has been Feros Abookadjie for Socket.dev. This is Casey Ellis for the Risky Business Podcast. Cheers.

Thanks, Casey. I appreciate it.