Scores of vulnerability found in experimental features can be too high
joyeecheung opened this issue · comments
I recently happened to look into the vulnerability reports since Node 10 to hound for "reasons to push infra teams in companies to invest in an upgrade" and noticed that quite a lot of vulnerabilities rated with high severity only have very limited impact. In particular, this applies to high severity vulnerabilities that are applicable only when users rely on policy or permissions, which are experimental features behind flags.
While I don't have data for the adoption of these features (nodejs/next-10#261 should help giving us some insights into this), I suspect the percentage of users who would be affected by those would be <1%; and in terms of what I was looking for, these vulnerabilities in experimental features probably aren't relevant enough for any company to invest in an infra upgrade, even when they are rated as high severity and look serious (most companies that count on the stability of Node.js simply won't rely on experimental features in production) . Take https://hackerone.com/reports/2199818 for example - this gets a 9.8 CVSS Score and rated as high severity, so it looked like a big deal, until you realize that it's nothing to worry about if you don't use --experimental-permission, and only very few people use it currently, AFAICT.
To me it doesn't seem quite right that vulnerabilities in features that the ecosystem doesn't generally rely on (yet) are rated the same way as vulnerabilities in features that are widely used. The over-rating can potentially lead to alarm fatigue and could lead to people taking the security updates less seriously overall.
I am not quite sure how we should adjust the process or how we can change the CVSS vector assessment for this, but at least the manual rating is something we can tweak. One immature suggestion could be, for vulnerabilities that only apply to experimental features used by <1% of our users (not sure if the survey data is reliable enough in this regard), or experimental features that are not yet reaching "release candidate" status, the manual rating of severity should be no more than "low". If the experimental feature is actually already used by many people, say >10%, or there are known popular npm packages with >100K downloads/week combined relying on it, then the upperbound can be "medium". These are all very imaginary numbers, I'd appreciate if someone more knowledgable about security rating can devise a more accurate model, but I do believe adjusting the rating according to feature adoption would help making sure people take the security updates more seriously.
Note that the CVSS does not have an indicator of “how many users are affected”… by design.
It’s a failure of that scoring system.
For reference, we have iterated over it in the past: nodejs/TSC#1299
TL;DR: I brought up that discussion in the past and turns out that regardless of ~1% of Node.js users make real use of it, this vulnerability is still HIGH for the ones who use it. We have decided to assess experimental features in the same way we assess stable features. Although it may seem more pressing than it actually is, I believe that it is the safer and more appropriate approach.
TL;DR: I brought up that discussion in the past and turns out that regardless of ~1% of Node.js users make real use of it, this vulnerability is still HIGH for the ones who use it. We have decided to assess experimental features in the same way we assess stable features. Although it may seem more pressing than it actually is, I believe that it is the safer and more appropriate approach.
I do agree, it's not easy to estimate the impact of a vulnerability, so we can only measure the severity
I see, if we cannot do anything about the severity, I wonder if we could at least make assessments of the vulnerabilities more effortless for the users. For example, we can try to develop a template for the descriptions, and consistently begin the description with something like:
Only users who rely on
--experimental-flagare affected by this vulnerability
Or, with an even stronger wording:
Users who do not currently rely on
--experimental-flagare not affected by this vulnerability.
instead of placing the scope near the end of the description. Or applying [Stable] [Experimental] [Legacy] label to the title according to the stability index of the feature being affected. That could already go a long way. With the current model, for people scrolling through the summaries to understand whether how much the vulnerability matters for them, I suspect this would probably be the most valuable information, as most of our vulnerabilities happen in very specific situations that only affect a minority of the audience, and one just want to know whether this matters for them or not before they even dive into how the vulnerability works.
For example, we can try to develop a template for the descriptions, and consistently begin the description with something like:
Well, we normally mention it in the security release notes, for instance:
"This vulnerability affects all users using the experimental permission model in active release lines: 20.x and 21.x."
Well, we normally mention it in the security release notes, for instance:
I think the key is to start the notes with "who won't be affected", not put them in the middle or near the end, so that users could just skip the rest when it doesn't apply to them. The current information template of the notes is more like
- there is a high severity vulnerability!
- explains how it works
- explain who are affected (at this point, user realize it doesn't matter for them, and this repeats for several vulnerabilities in a row)
I think that can more easily lead to alarm fatigue. A template like
- there is a high severity vulnerability in an experimental feature (users who don't use any experimental features skips)
- explain who are not affected (users who don't use this feature skips)
- explain how it works
may be more optimal in terms of reducing the fatigue, especially when they come out in a row.
Speaking of "who are affected" and "who are not" and a template for messaging, I also recall that after https://nodejs.org/en/blog/vulnerability/april-2024-security-releases-2 came out, someone reached out to me in private saying that the blog post should also mention that EOL releases are affected. While I am not sure if it's worth the trouble to investigate in detail the extent of impact on EOL releases, it's probably still useful to remind those who are still stuck with EOL releases that in general, EOL releases should be considered affected if they have the affected feature (by not mentioning EOL at all, or emphasizing on active release lines, it seems readers can get the wrong idea that EOL releases are somehow not affected).
someone reached out to me in private saying that the blog post should also mention that EOL releases are affected. While I am not sure if it's worth the trouble to investigate in detail the extent of impact on EOL releases, it's probably still useful to remind those who are still stuck with EOL releases that in general, EOL releases should be considered affected if they have the affected feature (by not mentioning EOL at all, or emphasizing on active release lines, it seems readers can get the wrong idea that EOL releases are somehow not affected).
It is important to note that End-of-Life versions (EOLs) are always considered impacted if a security release was made after they became EOL. You can find more information regarding this topic in #1025.
Regarding this particular issue, the only viable solution would be to create a new template for security release updates. So I believe this issue should be moved to https://github.com/nodejs-private/security-release and included in the automation process.