The criteria https://github.com/ossf/criticality_score/blob/main/README.m... are designed to give higher scores to projects that are updated frequently by many contributors from different organizations, so of course the random Nebraskan's critical project is going to get a low "criticality score".
I think we could improve it a bit. For example, Spring boot should have a very low score for me. It's backed by a large company Pivotal. They don't need any support I think. Same thing for elasticsearch.
For me:
- backed by a large company ?
- number of contributor doing 80% of the work ? or active in the last 12 months ? commits breakdown (99% is done by one guy) ?
- issues created/closed ratio
- PR created/merged ratio
- use critical projects ?
- other from your original score
A nice bonus: if we could use the tool to assess critical score for our project (not globally). For local dependency, we could increase the critical value if dependents count is low. Very few person is using it: that's a bad sign. With this, we could find those dependencies.
We could also create a global score (like you did) by using the previous score and scaling it using the dependency usage (dependents_count like you did).
With this calculation, I think it's more likely to find relevant projects.
How to find it's backed by a large company ? Not sure about this, we can check if the project is part of an organization, if contributor have a company or if they have a pro account. For example, if the top 5 contributors are from Google, it's likely it's sponsored by it(could be done during their free time but less likely).
Note: check what happens with a stable project (no new issue and PR).
I'm not sure if this is relevant, but bash and the readline lib are both maintained by a single unpaid volunteer. (I don't know if he's from Nebraska though.)
Gensim is #119 in the list according to your link, far behind projects with many more active contributors, so hardly a resounding success of your scoring method.
In terms of metrics, you could start by weighing projects with few contributors as more critical, not less. Specifically, gensim does appear to have had quite a few contributors, but the bulk of the code was written by the single maintainer https://github.com/RaRe-Technologies/gensim/graphs/contribut... So maybe you should add a metric "percentage of code in the past year authored by the top contributor".
If you want to go about it in a more data-driven fashion, you could go through the top projects for each language, check whether they actually need your support (e.g. find out what the development goals are, ask whether the current resources are sufficient and what they'd do with the additional resources you can provide) to get a ground-truth labeling of critical projects, then readjust your weights to match the ground truth.
I don't think it's explicit in the article, but that's a comic (XKCD). Relevant discussion [1] suggests that there's not a specific project referenced by the comic.
When I first read this comic, ntpd [2] [3] came to mind.
At least NTP is a standard protocol and there are a bunch of cromulent alternatives to the original Mills ntpd. Chrony and ntpsec are both reasonable. If any one of the implementations went away tomorrow, distros would scramble but they'd have a place to go.
I'm interested in the gulf between low-level systems code and effectively end-user code here. Things like GLib, Cairo, Harbuzz, etc -- none of the end-user tools work without them. It really demonstrates just how difficult this evaluation can be.
It is definitely difficult especially with critical dependencies. We are looking for any criteria to identify these in automated fashion. Parallely, for ones we know about those are we are trying to run our automated tools on. E.g. glib, cairo, harfbuzz are all continuously fuzzed as part of OSS-Fuzz - https://github.com/google/oss-fuzz/tree/master/projects
Please consider allowing scanning tarball/zip distributions of source directly as well. It is a SCM-agnostic method that is also well-supported by GitHub, Gitiles, hgweb, and many old but still-in-use projects that pre-date Git.
It would be nice if this could be mentioned a bit clearer in the blog post and/or README; it's not really that obvious at all and I had to go to the source to check, and loads of people here seem confused about it since it more or less implies "we looked at all open source projects".
The issue with the mirror is we don't get the important stats to make decisions. E.g. number of contributors, issue changes due to custom issue tracker. We are still thinking on how to add information from such cases in automated fashion, ideas welcome!
EDIT: It's really Custom Search Engine rebranded to me, it doesn't index all you website, I still prefer Algolia if I had to have a external search provider.
EDIT 2: Seems very customizable, maybe by adding all your urls it will index all you site, making it a viable alternative to Algolia
I also researched the best diff algorithm. Google's diff-match-patch [1] library produce very good diffs for example. But I found that the best diffs are produced by wikidiff2 [2], the MediaWiki diff engine. Both engines produce word-by-word diff.
