The Fault in Our Stars: An Analysis of GitHub Stars as an Importance Metric for Web Source Code
Open accessAbstract
Are GitHub stars a good surrogate metric to assess the importance of open-source code?While security research frequently uses them as a proxy for importance, the reliability of this relationship has not been studied yet.Furthermore, its relationship to download numbers provided by code registries -another commonly used metric -has yet to be ascertained.We address this research gap by analyzing the correlation between both GitHub stars and download numbers as well as their correlation with detected deployments across websites.Our data set consists of 925 978 data points across three web programming languages: PHP, Ruby, and JavaScript.We assess deployment across websites using 58 hand-crafted fingerprints for JavaScript libraries.Our results reveal a weak relationship between GitHub Stars and download numbers ranging from a correlation of 0.47 for PHP down to 0.14 for JavaScript, as well as a high amount of low star and high download projects for PHP and Ruby and an opposite pattern for JavaScript with a noticeably higher count of high star and apparently low download libraries.Concerning the relationship for detected deployments, we discovered a correlation of 0.61 and 0.63 with stars and downloads, respectively.Our results indicate that both downloads and stars pose a moderately strong indicator of the importance of client-side deployed JavaScript libraries.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The number of GitHub stars is widely used when stakeholders evaluate, select, and adopt open-source projects into their software supply chain.
“Many studies provided evidence that the number of GitHub stars is widely used when stakeholders evaluate, select, and adopt opensource projects into their software supply chain [33, 61, 71, 76, 89, 95].”
From Six Million (Suspected) Fake Stars on GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware (2026) · cites this paper as [61]
The Introduction says security researchers commonly use GitHub stars to select applications or libraries as research targets. It does not establish that stakeholders generally use stars to evaluate or adopt open-source projects into their software supply chains; the Background explicitly leaves the connection between stars and usage open. (LLM, conf=0.86)
Experimental — accuracy not yet measuredStars on GitHub are not a reliable signal.
“Our study is far from the first showing that stars on GitHub are not a reliable signal [2, 56, 61, 80].”
From Six Million (Suspected) Fake Stars on GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware (2026) · cites this paper as [61]
Section A, “Untrustworthy Connection Between Project and Repository” (page unavailable), says star counts for registry projects cannot be taken at face value. But Section C, “Correlations” (page unavailable), describes stars as moderately strong predictors of deployments for the client-side JavaScript libraries studied. The unqualified claim that GitHub stars are not a reliable signal overstates the paper’s findings. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThe number of GitHub stars is merely a popularity signal, with only moderate correlation to actual usage or importance.
“it is merely a popularity signal (i.e., the attention received from GitHub users), with only moderate correlation to actual usage or importance as shown in prior studies [56, 61, 80].”
From Six Million (Suspected) Fake Stars on GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware (2026) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- V. Le Pochat, T. van Goethem, S. Tajalizadehkhoob, M. Korczynski, and W. Joosen, "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation," in Network and Distributed System Security Symposium, 2019.
- M. Musch and M. Johns, "U can't debug this: Detecting javascript anti-debugging techniques in the wild." in USENIX Security Symposium, 2021.
- S. Khodayari and G. Pellegrino, "It's (DOM) Clobbering Time: Attack Techniques, Prevalence, and Defenses," in IEEE Symposium on Security and Privacy, 2023.
- J. Bosamiya, W. S. Lim, and B. Parno, "Provably-safe multilingual soft- ware sandboxing using webassembly." in USENIX Security Symposium, 2022.
- R. P. Kasturi, J. Fuller, Y. Sun, O. Chabklo, A. Rodriguez, J. Park, and B. Saltaformaggio, "Mistrust plugins you must: A large-scale study of malicious plugins in wordpress marketplaces." in USENIX Security Symposium, 2022.
- F. Xiao, J. Huang, Y. Xiong, G. Yang, H. Hu, G. Gu, and W. Lee, "Abusing hidden properties to attack the node.js ecosystem." in USENIX Security Symposium, 2021.
- J. C. Davis, E. R. Williamson, and D. Lee, "A sense of time for javascript and node.js: First-class timeouts as a cure for event handler poisoning." in USENIX Security Symposium, 2018.
- S. Koch, M. Wessels, D. Klein, and M. Johns, "Poster: The risk of insufficient isolation of database transactions in web applications," in ACM SIGSAC Conference on Computer and Communications Security, 2023.
- C. Zuo and Z. Lin, "Playing without paying: Detecting vulnerable payment verification in native binaries of unity mobile games." in USENIX Security Symposium, 2022.
- W. Li, J. Ming, X. Luo, and H. Cai, "Polycruise: A cross-language dynamic information flow analysis." in USENIX Security Symposium, 2022.
- Y. Li, Y. Sun, Z. Xu, J. Cao, Y. Li, R. Li, H. Chen, S.-C. Cheung, Y. Liu, and Y. Xiao, "Regexscalpel: Regular expression denial of service (redos) defense by localize-and-fix." in USENIX Security Symposium, 2022.
- D. Klein and M. Johns, "Parse Me, Baby, One More Time: Bypassing HTML Sanitizer via Parsing Differentials," in IEEE Symposium on Security and Privacy, 2024.
- G. Pellegrino, M. Johns, S. Koch, M. Backes, and C. Rossow, "Deemon: Detecting CSRF with Dynamic Analysis and Property Graphs," in ACM SIGSAC Conference on Computer and Communications Security, 2017.
- S. Koch, T. Sauer, M. Johns, and G. Pellegrino, "Raccoon: Automated verification of guarded race conditions in web applications," in ACM Symposium on Applied Computing, 2020.
- K. Du, H. Yang, Y. Zhang, H. Duan, H. Wang, S. Hao, Z. Li, and M. Yang, "Understanding promotion-as-a-service on github," in Annual Computer Security Applications Conference, 2020.
- D. Pinckney, F. Cassano, A. Guha, and J. Bell, "Npm-Follower: A Complete Dataset Tracking the NPM Ecosystem," in ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023.
- V. Kotu and B. Deshpande, "Chapter 4 -classification," in Data Science (Second Edition), second edition ed. Morgan Kaufmann, 2019.
- G. J. Székely, M. L. Rizzo, and N. K. Bakirov, "Measuring and testing dependence by correlation of distances," The Annals of Statistics, 2007.
- M. Zimmermann, C.-A. Staicu, C. Tenny, and M. Pradel, "Small world with high risks: A study of security threats in the npm ecosystem." in USENIX Security Symposium, 2019.
- B. Dolan-Gavitt, P. Hulin, E. Kirda, T. Leek, A. Mambretti, W. K. Robertson, F. Ulrich, and R. Whelan, "LAVA: Large-Scale Automated Vulnerability Addition," in IEEE Symposium on Security and Privacy, 2016.
- M. Schloegel, N. Bars, N. Schiller, L. Bernhard, T. Scharnowski, A. Crump, A. Ale-Ebrahim, N. Bissantz, M. Muench, and T. Holz, "SoK: Prudent Evaluation Practices for Fuzzing," in IEEE Symposium on Security and Privacy, 2024.
- H. Borges, A. Hora, and M. T. Valente, "Understanding the factors that impact the popularity of github repositories," in IEEE International Conference on Software Maintenance and Evolution, 2016.
- K. Aggarwal, A. Hindle, and E. Stroulia, "Co-evolution of project documentation and popularity within github," in International Conference on Mining Software Repositories, 2014.
- D. Syed, J. Sessa, A. Henschel, and D. Svetinovic, "Data analysis of correlation between project popularity and code change frequency," in Neural Information Processing, 2016.
- A. Zerouali, T. Mens, G. Robles, and J. M. Gonzalez-Barahona, "On the diversity of software package popularity metrics: An empirical study of npm," in IEEE International Conference on Software Analysis, Evolution and Reengineering, 2019.
Cited by
Sorted by citations
He, H., Yang, H., Burckhardt, P., Kapravelos, A., Vasilescu, B., & Kästner, C. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764531 · Cited by 2 · see citing works
- Authors
- Koch, S., Klein, D., & Johns, M.
- Affiliations
- Technische Universität Braunschweig (DE)
- Venue
- Proceedings 2024 Workshop on Measurements, Attacks, and Defenses for the Web
- Publisher
- Internet Society
- Published
- 2024
- OpenAlex
- W4407278772
- Cited by
- 8
- References
- 0
- Citations by year
- 2026: 4 · 2025: 4
- FWCI
- 3.04
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Workshop on Measurements, Attacks, and Defenses for the Web (MADWeb) (publishedVersion)
- Funders
- European Commission (HORIZON2020, 101019206)
- Deutsche Forschungsgemeinschaft (390781972)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- open source software, JavaScript library, PHP, ruby, web application security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
