Rendered at 00:52:45 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ashishb 7 hours ago [-]
Hypothesis: a big accelerant of these rapid repository compromise (from Red hat to GitHub to Amazon to small startups) might be GitHub+dependabot automatic dependency updates.
So, just like COVID-19 used air travel, modern malware attacks are relying on GitHub+dependabot to speed up the spread.
Even for single page website built using Vue, I would get about 5 updates a week.
woodruffw 3 hours ago [-]
It’s plausible. It’s certainly the case that we (meaning security practitioners) spent years trying to move people onto faster and more automated update cycles, and these kinds of compromises have revealed a latent weakness that comes wit doing so.
swader999 10 hours ago [-]
Aren't we back to the drawing board once everyone uses this?
grncdr 10 hours ago [-]
I think the idea is that dedicated security firms and/or automated scanners will discover exploits in the cooldown period.
woodruffw 8 hours ago [-]
Yep, this is the thesis behind them. I wish people engaged more fully with this argument: it’s possible to believe that security vendors won’t do a good job of upholding their side of the bargain, but I’ve yet to see anybody argue that rather than making a faulty universalization argument against cooldowns.
mike_hock 3 hours ago [-]
Anything that isn't blatantly obvious zero effort malware is gonna wait for the cooldown to expire and then gradually introduce backdoors or vulnerabilities in subsequent versions once the project has gained trust.
woodruffw 3 hours ago [-]
I think you’re talking about something different than the median case here: the kinds of malware that cooldowns are effective against are attacking already trusted projects, e.g. via repository compromise.
I think it remains to be seen whether the economics of this kind of “supply chain” compromise support a stealthier actor profile: the whole reason to compromise an OSS project rather than exploit a single victim’s browser or phone is to smash-and-grab en masse. That’s a fundamentally noisy and delay-sensitive attacker profile.
weinzierl 7 hours ago [-]
If this is the idea, why don't we let the dedicated security firms and/or automated scanners find the vulnerabilities before the release?
You need an early release in the "given enough eyeballs all bugs are shallow" world because you need the eyeballs, but if you count on specialists and scanners no general availability release is necessary and hence no cool down.
nixpulvis 4 hours ago [-]
Because this isn't about guarding against general vulnerabilities. This is about guarding against a class of supply chain attack where an attacker compromises the packaging system in a way that they slip in a malicious release.
It is true if everyone waits a cooldown (I don't like this name, but that's another thing) period, then it doesn't solve anything, but as others have pointed out, just because you don't build off the cooldown doesn't mean you and others aren't watching the releases.
Closing the window makes the burden of hiding the exploit higher, which is I think objectively an improvement of security posture.
john_strinlai 6 hours ago [-]
i am not sure what the benefits of your proposal are compared to the "cooldown period" way.
the releases will be delayed for the same time period, but you increase the amount of coordination required significantly and reduce user agency.
weinzierl 4 hours ago [-]
Currently we release software to the public early and let people find vulnerabilities after release. A few early updaters get burned but the rest of us is saved. This cannot go on.
One idea to get out of this mess is to use cooldowns. But this idea only works if what GP said is true: Vulnerabilities are mostly found by (paid) specialists and advanced tools and not the general user populations.
My point is that, if that is true we do not need to release insecure software. Do the magic before the release and no one gets hurt.
AgentME 3 hours ago [-]
We could rename "cooldown" to "pre-release period".
password4321 10 hours ago [-]
The point is to allow the automated scanners a chance to run.
Every security company and their cousin wants to be the one to find the next big dependency malware.
x0x0 7 hours ago [-]
The idea that a package can be updated and with a deploy at the right time could be live on your servers in prod 10 minutes later has always been crazy, and the last years have just reinforced that.
People are encouraged by package managers to treat any bit of code someone tosses onto a package manager as equivalent in reliability to the core language and sdk.
ihumanable 10 hours ago [-]
Yea, all the new advice around using dependency cooldowns only works if _someone_ is installing these things before you and finding the vulnerabilities.
It seems like the advice right now is to become a freerider while there are still people installing closer to release that will do free work for you finding out there's something nasty in the release.
Once everyone is waiting 2 weeks to install an update, then the value of everyone waiting goes down dramatically.
postalcoder 8 hours ago [-]
Using dependency cooldowns is not a free-rider problem. There's a real tradeoff here – ppl are trading their time preference for security.
Just as users are incentivized to avoid malware, researchers and attackers are equally motivated to be the first to discover it.
The concern trolling around widespread dependency cooldowns doesn't make sense. Most people shouldn't be eager to download a release that hasn't made its way through at least some scans.
kbenson 10 hours ago [-]
This is how a chunk of people function anyway. There are plenty of people that choose to not install "point zero" release for software of a certain importance, assuming with any major changes there are often bugs that come along with it.
In this case, since the number of cool down days is configurable, even if everyone was using it we would still likely see a somewhat smooth curve for adoption, since not everyone will choose the same delay and the delay time will likely map closely to how people want to habdke risk.
It's all a trade off, just like it's always been. This just makes it simpler to act on what you want your risk/comfort level to be.
raesene9 9 hours ago [-]
not really, there are a number of security companies doing analysis of any new packages looking for supply chain attacks, so if you wait a couple of days, till their analysis is complete, you're reducing the risk of hitting a compromised package.
teeray 10 hours ago [-]
It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
embedding-shape 8 hours ago [-]
> It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
Except there is lots to gain from being the first to write about the new malware on some registry, so companies are actively downloading and inspecting literally every package.
Back in the day (maybe 6-7 years ago?) you could detect this by uploading a new npm package that hit back some endpoint in your control, and it was almost guaranteed that this endpoint got a request within a minute of publishing a new package or update to existing one with users. Nowadays I think none of the scanners actually run the code, mostly static-analysis, and I dunno how often the npm download counter updates per day, probably harder to see in real-time.
teeray 8 hours ago [-]
> Except there is lots to gain from being the first to write about the new malware on some registry
Show me the company writing to their customers “we intentionally decided to ship code with potentially novel vulnerabilities. One of those vulnerabilities caused disclosure of your data, but cheer up! We have this cool security blog post about it now.” Meanwhile their competitors freeride and their customers’ data is safe.
scheme271 6 hours ago [-]
I think it's more some security company writing about a vulnerability they discovered in this module or a worm/backdoor and not the company that wrote the software. The security company gets publicity and potentially gets more biz for security consulting.
weaksauce 7 hours ago [-]
security researchers not the ones shipping the faulty code.
teeray 7 hours ago [-]
We’re not talking about security researchers here:
> there is lots to gain from being the first to write about the new malware on some registry, so *companies* are actively downloading and inspecting literally every package.
(Emphasis mine)
weaksauce 5 hours ago [-]
yeah security researchers at security companies are the ones we are talking about.
john_strinlai 6 hours ago [-]
>We’re not talking about security researchers here:
we are.
"companies" in this context is "security companies" (hence why they are "downloading and inspecting every package", which would not make sense if referring to the people authoring and shipping a single package)
tancop 7 hours ago [-]
[dead]
delichon 11 hours ago [-]
> A version whose source does not expose created_at, such as older gem servers, historical entries from before the v2 cutover, or private registries still on the v1 format, is treated as outside the window and stays resolvable.
How is that not an easy exploit to circumvent the cooldown?
werdnapk 10 hours ago [-]
Most gems in Ruby/Rails projects come from rubygems, so if they were published long ago, any exploits should have already been found hopefully. Any old gems that would attempt to release a new compromised version would now get a created_at timestamp and the cooldown applies.
Unless you can compromise the gem server to overwrite created_at fields, I don't see any exploits here.
Private gem servers are either already trusted (if they're your own) or already under some scrutiny and extra care already being taken (ideally), but this last case applies to very few projects I'm sure.
tenderlove 6 hours ago [-]
RubyGems.org backfilled older releases with `created_at` fields, so theoretically you could still do the cooldown with very old gems (though I don't know why you would). It's only private / alternative gem servers that may not provide `created_at` fields.
OptionOfT 10 hours ago [-]
Can you in your own gem depend on gems from another server? Or does it need to be configured on the client?
If not, and the current defacto standard gem server doesn't accept v1 anymore, we're good I suppose?
doctorpangloss 9 hours ago [-]
you have 1.0 installed. you enable 7 day cooldowns. an exploit is discovered in 1.0, and 1.1 is immediately released to fix the exploit. do you sit on 1.0 for 7 days?
trevor-e 9 hours ago [-]
it specifically addresses this in the "The escape hatch" section...
k3nx 7 hours ago [-]
So, the threat actor now, after making the compromise, just needs to announce that the previous version has a 0-day, and folks need to install the latest version? I love the idea of a cool down, but it can still be thwarted. I would just hope folks that are trying to patch a 0-day take extra caution to vet the new version. I wouldn't be opposed to a --cooldown 0 doing a side by side diff. I may not know what's going on in the code, but a 0-day shouldn't be a ton of new code either.
esafak 8 hours ago [-]
Security updates bypass the cooldown.
doctorpangloss 2 hours ago [-]
But what channel decides it is a security update? How do you know? Someone has to notify whom exactly? And what if the adversary says their supply chain attack commit is a security update?
All of this cooldown stuff is so mind bogglingly stupid...
9 hours ago [-]
shevy-java 7 hours ago [-]
Meanwhile ruby is dropping ranks.
How active is rubygems.org itself? I retired when the 100k download threshold was installed onto developers there; on github I don't have any such restriction pertaining to code I publish and maintain. But even before that restriction, numerous gems were abandoned. I understand that this is a natural cycle anyway, but without an influx of new developers, ruby will fossilize and age out just as perl did before.
None of those "cooldowns" will bring in new developers either. It all seems to be about meta-appeasing companies; this could indirectly help, but I doubt it will help much.
So, just like COVID-19 used air travel, modern malware attacks are relying on GitHub+dependabot to speed up the spread.
Even for single page website built using Vue, I would get about 5 updates a week.
I think it remains to be seen whether the economics of this kind of “supply chain” compromise support a stealthier actor profile: the whole reason to compromise an OSS project rather than exploit a single victim’s browser or phone is to smash-and-grab en masse. That’s a fundamentally noisy and delay-sensitive attacker profile.
You need an early release in the "given enough eyeballs all bugs are shallow" world because you need the eyeballs, but if you count on specialists and scanners no general availability release is necessary and hence no cool down.
It is true if everyone waits a cooldown (I don't like this name, but that's another thing) period, then it doesn't solve anything, but as others have pointed out, just because you don't build off the cooldown doesn't mean you and others aren't watching the releases.
Closing the window makes the burden of hiding the exploit higher, which is I think objectively an improvement of security posture.
the releases will be delayed for the same time period, but you increase the amount of coordination required significantly and reduce user agency.
One idea to get out of this mess is to use cooldowns. But this idea only works if what GP said is true: Vulnerabilities are mostly found by (paid) specialists and advanced tools and not the general user populations.
My point is that, if that is true we do not need to release insecure software. Do the magic before the release and no one gets hurt.
Every security company and their cousin wants to be the one to find the next big dependency malware.
People are encouraged by package managers to treat any bit of code someone tosses onto a package manager as equivalent in reliability to the core language and sdk.
It seems like the advice right now is to become a freerider while there are still people installing closer to release that will do free work for you finding out there's something nasty in the release.
Once everyone is waiting 2 weeks to install an update, then the value of everyone waiting goes down dramatically.
Just as users are incentivized to avoid malware, researchers and attackers are equally motivated to be the first to discover it.
The concern trolling around widespread dependency cooldowns doesn't make sense. Most people shouldn't be eager to download a release that hasn't made its way through at least some scans.
In this case, since the number of cool down days is configurable, even if everyone was using it we would still likely see a somewhat smooth curve for adoption, since not everyone will choose the same delay and the delay time will likely map closely to how people want to habdke risk.
It's all a trade off, just like it's always been. This just makes it simpler to act on what you want your risk/comfort level to be.
Except there is lots to gain from being the first to write about the new malware on some registry, so companies are actively downloading and inspecting literally every package.
Back in the day (maybe 6-7 years ago?) you could detect this by uploading a new npm package that hit back some endpoint in your control, and it was almost guaranteed that this endpoint got a request within a minute of publishing a new package or update to existing one with users. Nowadays I think none of the scanners actually run the code, mostly static-analysis, and I dunno how often the npm download counter updates per day, probably harder to see in real-time.
Show me the company writing to their customers “we intentionally decided to ship code with potentially novel vulnerabilities. One of those vulnerabilities caused disclosure of your data, but cheer up! We have this cool security blog post about it now.” Meanwhile their competitors freeride and their customers’ data is safe.
> there is lots to gain from being the first to write about the new malware on some registry, so *companies* are actively downloading and inspecting literally every package.
(Emphasis mine)
we are.
"companies" in this context is "security companies" (hence why they are "downloading and inspecting every package", which would not make sense if referring to the people authoring and shipping a single package)
How is that not an easy exploit to circumvent the cooldown?
Unless you can compromise the gem server to overwrite created_at fields, I don't see any exploits here.
Private gem servers are either already trusted (if they're your own) or already under some scrutiny and extra care already being taken (ideally), but this last case applies to very few projects I'm sure.
If not, and the current defacto standard gem server doesn't accept v1 anymore, we're good I suppose?
All of this cooldown stuff is so mind bogglingly stupid...
How active is rubygems.org itself? I retired when the 100k download threshold was installed onto developers there; on github I don't have any such restriction pertaining to code I publish and maintain. But even before that restriction, numerous gems were abandoned. I understand that this is a natural cycle anyway, but without an influx of new developers, ruby will fossilize and age out just as perl did before.
None of those "cooldowns" will bring in new developers either. It all seems to be about meta-appeasing companies; this could indirectly help, but I doubt it will help much.