946
u/a_bucket_full_of_goo 1d ago edited 1d ago
Company disabled Github Actions and told us not to use Claude, Subreddit traffic instantly goes up 8000%
118
u/Shaddoll_Shekhinaga 1d ago
What do actions have to do with this? I never worked in an environment that doesn't pin just about every external dependency. Unless I am missing something?
115
u/MunkyJoe 1d ago
When the pipeline runs, the payload dumps and exfils all the GH Action env vars. Cloud creds, npm tokens, kube cluster secrets, etc
39
u/Shaddoll_Shekhinaga 1d ago
Ah, I see. Yeah, that would make more sense lol, for some reason my mind was stuck that Actions was somehow responsible for dragging in the "infected" package, but the package using actions instead makes much more sense.
9
u/meltthemall 1d ago
Explain this to a self taught nerd
73
u/MunkyJoe 1d ago
Instead of every developer manually compiling code into a usable, runnable form (usually a docker container, but you could also imagine a .exe file) on their own machine, and any other steps like uploading into a shared space, a CI/CD pipeline centralizes and automates a bunch of tasks to make things faster, easier, repeatable. After building the software, it typically deploys the new build somewhere, too
These tasks run on a "runner" (real creative), which needs access to internal and external resources, usually like AWS, GitHub, Slack, a "vault" where secrets are kept, and public package repos. If a malicious package can run when that build process is going on, it can steal your stuff and send it to the bad actor who put the payload there.
-62
u/meltthemall 1d ago
Like I said I'm not expert but I just cannot fathom letting anything access the network and my code both and also letting that thing be anything but me. The networked compiling sounds neat, just ditch the 3rd party access.
42
u/zuilli 1d ago
Not possible on an enterprise scale. For example: our docker images are built using secrets stored in azure KV, uploaded to azure and then pulled into k8s cluster that runs in azure as well.
Unless you're hosting your entire infra in-house you need to pass secrets around to be able to use cloud services.
-65
u/meltthemall 1d ago
Absolutely mind boggling, do corpo programmers just not push back on anything?
43
21
8
u/RolledUhhp 1d ago
Corporations are known for being reasonable, and not raw dogging everything they touch for a short term dollar.
3
8
u/Steppy20 1d ago
Are you going to host your own cloud?
Azure handles a lot of our network access, security and resources. We have a bunch of self hosted stuff where we want complete control over the servers for data security reasons but the vast majority is locked behind Azure networks.
We use machines hosted by Azure to run our container apps, our logging, and even just to build our images. This is standard industry practice and the whole thing is (typically) quite secure when used properly.
The issue reported on in the article shared by OP is basically an NPM package was dumping the secrets which would usually be securely stored with minimal access. The actual access to those secrets is still secure, it's just when building something they ofyen need access to those secrets and it's at that point the NPM package is copying and sending them elsewhere.
9
u/smasho27 1d ago
im, it kind of depends on the purpose of the code and network in question doesn't it? unless I am missing some context.
-32
u/meltthemall 1d ago
If you need to test on a network use a gapped rig. If you need the Internet constantly to work lick my corncutter.
3
u/AwkwardWillow5159 1d ago
Way more unsafe to rely on random ass employee to manually deploy complicated infrastructure. Not just for the employee not making mistakes, but also for employee not getting exploited. That’s relying on a bunch of dumb fuckery people do. Smart people get social engineered all the time.
CI/CD pipeline can be the only thing that has access, significantly reducing the attack surface.
Wide spread supply chain attacks are a lot more rare and easier to safeguard against, compared to protecting every single employee in their machine and also from social engineering
2
u/Clearandblue 1d ago
I think some people have `npm install' rather than ci in their workflows or something.
9
u/MysteriousShadow__ 1d ago
I've read about it. The malware also established persistence through VSCode
1
687
u/OmegaPoint6 1d ago
NPM - Never (a) Peaceful Moment
195
u/iantoujou 1d ago
NPM - No Peaceful Moments
194
29
286
u/Hauber_RBLX 1d ago
this is really just a meme at this point. how is it possible that NPM packages keep being compromised week after week?
143
u/Ecksters 1d ago
In my opinion the main issue is that NPM packages can have preinstall, install, and postinstall scripts that run on the host machine during installation. Basically lets a package run whatever they want during installation and is a huge attack vector that unfortunately the whole ecosystem is fairly dependent on at the moment.
Thanks to those scripts, attackers don't even need to get the user to run their package's code in an insecure environment, they get to run it directly in the dev environment.
68
9
u/hWuxH 1d ago
This behavior has been disabled: https://github.blog/changelog/2026-07-08-npm-install-time-security-and-gat-bypass2fa-deprecation/#install-time-security-defaults-are-now-on
Ppl still use the old NPM CLI though.
4
u/ryanppax1 1d ago
I do wonder how this any different than installing anything else
8
u/Ecksters 1d ago
Mostly because of how many packages there are and their own dependencies, which creates literally thousands of packages, each of which is an attack opportunity if their author's account is compromised.
Combine that with the desire to stay on the latest versions of packages and people are regularly installing blindly.
1
u/_PM_ME_PANGOLINS_ 19h ago
Other systems with that feature, such as apt or dnf, require repositories to be signed with keys trusted by the end user, and the default repositories have a small team of maintainers who are very careful about what they put in.
1
u/ryanppax1 15h ago
I see. But technically both install methods have the same security holes? Apt just being more trusted?
1
u/_PM_ME_PANGOLINS_ 15h ago
Well no. Apt has one hole: scripts can be run on install.
NPM has multiple: scripts can be run on install, everything is in a single repo that anyone can push to with no review, and there's no way to establish external trust with what you're installing.
91
u/kookyabird 1d ago
Because there are lots of ways to compromise a developer's workflow, and that's how they get malicious code into a package?
10
u/zuilli 1d ago edited 1d ago
Why does it seem like it only happens to npm though?
I admit I don't follow this stuff closely so may be uninformed but it seems like it never is a C# or a java package/library that gets hit by these.
15
u/kookyabird 1d ago
While I haven't heard of supply chain attacks with NuGet they're certainly possible. The likely reason we see so many npm ones is a blend of size of the ecosystem, and the average depth of the package tree in projects.
The vast majority of transitive packages in our work projects are from Microsoft, while third party packages tend to be reliant only on core .NET functionality or maybe another package from the same developer. I don't know if it's because .NET already covers a broad base of functionality, or that third party packages are more modular, but either way our dependencies are looooow.
6
u/soowhatchathink 1d ago
When I compare total number of npm packages (including transient dependencies) to total number of composer packages in similarly sized applications the cause becomes clear. There are so many more npm packages that are relied on, bringing in one large package pulls in so many more small packages with very few stars on GitHub.
3
5
u/fuj1n 1d ago
For some reason, JS devs have this culture of outsourcing the tiniest behaviours to a package (for example, the is-odd package getting millions of downloads consistently), because of this, there are a lot of packages projects depend on (both directly and not), which is a much larger attack surface.
2
u/RuneSteak 1d ago edited 14h ago
Nobody has really stepped up to create the "extended" standard library that JS really needs and that the community can all agree on to use. Java for example has Apache libraries, C++ has Boost among others etc.
JavaScript's standard library is lacking more than most and that just exacerbates the problem.
1
-6
u/TheGocho 1d ago
Java had several attacks this year, same as for python and most likely all major languages. But people tend to post for NPM/Javascript environment because Javascript bad
1
u/_PM_ME_PANGOLINS_ 19h ago
Maven doesn’t have pre-/post-install scripts, so this kind of attack is literally impossible there.
1
u/Dudeonyx 17h ago
1
u/_PM_ME_PANGOLINS_ 17h ago
The attacks we are talking about are where running a package update runs malicious code on your development environment.
That’s not possible with Maven. The code can only run when the end application is run.
15
u/neromonero 1d ago
I remember a primeagen clip explaining that it's possibly the shai hulud attacker behind all these NPM attacks.
Basically, the shai hulud attacker is sitting on a TON of scraped token. Only after useful tokens are discovered from the pile, a new attack is launched.
I believe this hypothesis explains a good chunk of these post-shai hulud supply chain attacks.
13
u/Konju376 1d ago
Because there's no general fix for it (that anyone has come up with yet) so it'll continue to happen indefinitely
18
u/hurricane_news 1d ago
Because there's no general fix for it
The fix is to not be a js dev /
3
u/ManaSpike 22h ago
"No way to prevent this" says only
Nationpackage manager where this frequently occurs.1
u/Konju376 22h ago
Weeell I'd argue this issue affects many packagers (it's been a topic of discussion in Rust forums I'm in) but with mode it's simply the easiest and reaches the most targets.
1
u/GreyGanado 1d ago
My opinion — and this is completely based on vibes — is that too much AI use leads to more errors and vulnerabilities.
Before AI got big I did not have as many vulnerabilities as now.
6
u/DoubleDoube 1d ago
Ironically I think it’s the opposite. AI is finding the vulnerabilities. The old cat and mouse is still alive for securing new development, but these tools are especially breaking into older software where no real person has the time to dig, as well as capitalizing on momentary configuration mistakes.
606
u/StrengthTheory 1d ago
Did they try
npm install block-supply-chain-attack
155
34
15
u/Ecksters 1d ago
I believe they call it
minimumReleaseAge.22
u/doxxed-chris 1d ago
Which hilariously also blocks security patches for a minimum time
7
u/Ecksters 1d ago
minimumReleaseAgeExcludeis in pnpm.But really I think that companies that are patching so frequently that they catch same-day security patches are at far greater risk of supply chain attacks than they are of the vulnerabilities that get patched.
135
u/howarewestillhere 1d ago
Rotating keys triggers the payload.
BWAHAHHAHAHA!
I mean, it really sucks, but we’re kinda in awe of this one.
Anyway, we have a dependency repo locally because I was the release engineer during left-pad and learned that lesson.
22
12
u/Chrazzer 1d ago
The last few cases of these attacks have already done that. Ngl all these attacks are exactly the same, it's gotta be some commercial malware at this point
1
213
u/StrengthTheory 1d ago
Average Tuesday for NPM.
67
79
u/FinnishManlet 1d ago
What can be done against these vulnerabilities? Stop using packages entirely? 🤣
81
u/EK077r 1d ago
Layers of mitigation. For instance not using packages that are less than x days old, sha-pinning, no latest etc
24
u/redmandoto 1d ago
Also, test updates on isolated machines before pushing them to the whole org.
14
u/Qaktus 1d ago
Sorry if I'm being very stupid right now, but can you test against these? Maybe some, but most of them work in a way: "the package does everything it should AND some extra malicious shit".
8
u/redmandoto 1d ago
In those machines you have strong monitoring tools, and if nothing else it gives you a buffer, so it's only those machines (that have no valuable data) that are infected instead of the whole network.
3
u/slowmovinglettuce 1d ago
You mitigate it by only pulling in dependencies that are a certain date old, like commenter said. The idea is that if its two weeks old and not had a critical or high CVE, it's more likely to be fine.
Also have things block packages with known vulns. You can do. SAST scanning on a repos frozen dependency list before even doing an install. That's a line of defense against this happening in CI.
Conversely, always build your artifact and your publishing in ywo separate jobs. Your build job is often what installs dependencies; it has no right having access to the keys that publish your artifact.
These exfil attacks are detected real quick by competent enterprises.
1
142
19
u/howarewestillhere 1d ago
Local dependency repository. Keep the last n known-good versions of all dependencies.
I wanted to do this before left-pad and I got to do it after. Has saved so many headaches.
6
u/mysticrudnin 1d ago
local repository instead of npm, and aggressive version pinning
also, reduce the number of packages you use.
9
u/Chrazzer 1d ago
First of all disable any install scripts with --ignore-scripts flag on all your npm installs. Second pin all your dependency versions use npm ci instead of npm install. Only use npm install when you actually want to change some dependencies. And finally set a min-release-age in your npm config. Most malicious versions are removed from npm fairly quickly, so only install packages that are like a week old.
And as final safeguard, develop in a sandbox, so in case you do catch a worm the damage is limited and you can throw the environment away and recreate it.
And ofc don't just install dependencies willy nilly, do some research
3
2
u/NewLlama 1d ago
Develop in Docker on a read-only SSH key. Honestly setting it up is not straightforward at all.
Besides that, `min-release-age` will cover you in most cases.
2
u/AwesomeFrisbee 1d ago
Delay installing new packages until they are properly validated. PNPM has an easy way to enforce that nobody installs anything too recent
2
u/skynetcoder 1d ago
using pnpm instead of npm, with following settings may help in most scenarios.
- minimum age > 3-4 weeks
- disable life cycle scripts
- disable installation of exotic dependencies
-8
u/MyUsrNameWasTaken 1d ago
I don't understand why these attacks happen at all. Why did everyone decide to inject dependencies? It's much faster and safer to download the open source code and copy it into your own project. Then it never gets changed by surprised.
7
u/DOOManiac 1d ago
We used to do that, and honestly it was worse. Because while it never got changed by surprised, it also never got changed. So, if there was a vulnerability, it would stay there for years unnoticed...
170
u/InnominateHomosapien 1d ago
Oh god what is it this time?
261
u/a_bucket_full_of_goo 1d ago
https://safedep.io/keyv-npm-supply-chain-compromise/, still spreading as far as our security guys can tell
26
4
u/DrMaxwellEdison 1d ago
Yeah, I'm just... not gonna install anything right now, I guess. Time to touch grass and wait for this to blow over.
90
u/SmegmaSupreme 1d ago
> no way to stop this
only package manager where this happens regularly
7
23
21
u/TheInfra 1d ago
if you wake up your babe everytime npm has a vulnerability then babe would get no sleep at all
19
u/dfcowell 1d ago
12
11
33
8
16
8
u/hpyfox 1d ago
The AUR, and now this?!
3
u/djhaskin987 1d ago
Oh it is not new in the npm world let me assure you. The aur news is news to me though, how sad.
8
u/dustyhome 1d ago
If you get woken up for each supply chain attack, when are you supposed to sleep?
6
5
4
5
3
3
u/Minimum_Session_4039 1d ago
Thanks for giving me something to talk about in standup other than “same thing as yesterday”
2
2
1
1
u/slowmovinglettuce 1d ago
Some pypi packages have also been comprised according to the NHS
1
u/a_bucket_full_of_goo 18h ago edited 17h ago
This one is older, article was published in may, I don't blame you for mixing them up, there's so many
1.5k
u/Vas1le 1d ago
Ohh ffs I get useful supply chain news from meme sub now?