The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature
Abstract
Software is increasingly crucial to scholarship, yet the visibility and usefulness of software in the scientific record are in question. Just as with data, the visibility of software in publications is related to incentives to share software in reusable ways, and so promote efficient science. In this article, we examine software in publications through content analysis of a random sample of 90 biology articles. We develop a coding scheme to identify software “mentions” and classify them according to their characteristics and ability to realize the functions of citations. Overall, we find diverse and problematic practices: Only between 31% and 43% of mentions involve formal citations; informal mentions are very common, even in high impact factor journals and across different kinds of software. Software is frequently inaccessible (15%–29% of packages in any form; between 90% and 98% of specific versions; only between 24%–40% provide source code). Cites to publications are particularly poor at providing version information, whereas informal mentions are particularly poor at providing crediting information. We provide recommendations to improve the practice of software citation, highlighting recent nascent efforts. Software plays an increasingly great role in scientific practice; it deserves a clear and useful place in scholarly communication.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Free online availability substantially increases a paper's impact
Lawrence, S. (2001) · Nature · doi:10.1038/35079151 · Cited by 797
Stodden, V., Guo, P., & Ma, Z. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067111 · Cited by 257
A New Approach for Scientific Citation Classification Using Cue Phrases
Pham, S. B., & Hoffmann, A. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24581-0_65 · Cited by 42
Cited by
Weaver, S., Shank, S. D., Spielman, S. J., Li, M., Muse, S. V., & Kosakovsky Pond, S. L. (2018) · Molecular Biology and Evolution · doi:10.1093/molbev/msx335 · Cited by 983
Pan, X., Yan, E., Cui, M., & Hua, W. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.03.005 · Cited by 491
PACVr: plastome assembly coverage visualization in R
Gruenstaeudl, M., & Jenke, N. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-3475-0 · Cited by 341
Smith, A. M., Katz, D. S., Niemeyer, K. E., & FORCE11 Software Citation Working Group (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.86 · Cited by 276
Investigating Software Usage in the Social Sciences: A Knowledge Graph Approach
Schindler, D., Zapilko, B., & Krüger, F. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-49461-2_16 · Cited by 24
Metadata
- Authors
- Howison, J., & Bullard, J.
- Venue
- Journal of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2015-05-13
- Volume
- 67
- Issue
- 9
- Pages
- 2137-2155
- ISSN
- 2330-1635 (ISSN-L)
- 2330-1643
- OpenAlex
- W1622429627
- Cited by
- 159
- References
- 5
- FWCI
- 13.46
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. https://app.cochat.ai/papers/10.1002/asi.23538
- The paper
- Howison, J., & Bullard, J. (2016). Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.23538
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.