Measuring the quality of scientific references in Wikipedia: an analysis of more than 115M citations to over 800 000 scientific articles
Open accessPublished version of a preprint
Abstract
Wikipedia is a widely used online reference work which cites hundreds of thousands of scientific articles across its entries. The quality of these citations has not been previously measured, and such measurements have a bearing on the reliability and quality of the scientific portions of this reference work. Using a novel technique, a massive database of qualitatively described citations, and machine learning algorithms, we analyzed 1 923 575 Wikipedia articles which cited a total of 824 298 scientific articles in our database and found that most scientific articles cited by Wikipedia articles are uncited or untested by subsequent studies, and the remainder show a wide variability in contradicting or supporting evidence. Additionally, we analyzed 51 804 643 scientific articles from journals indexed in the Web of Science and found that similarly most were uncited or untested by subsequent studies, while the remainder show a wide variability in contradicting or supporting evidence.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A number of researchers have already made use of scite for quantitative assessments of the literature.
“A number of researchers have already made use of scite for quantitative assessments of the literature.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Wakefield, A., Murch, S., Anthony, A., Linnell, J., Casson, D., Malik, M., Berelowitz, M., Dhillon, A., Thomson, M., Harvey, P., et al. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11096-0 · Cited by 3,006 · see citing works
RetractedCleavage of Structural Proteins during the Assembly of the Head of Bacteriophage T4
LAEMMLI, U. K. (1970) · Nature · doi:10.1038/227680a0 · Cited by 249,961 · see citing works
Raza, H., John, A., & Nemmar, A. (2013) · Cellular Physiology and Biochemistry · doi:10.1159/000350087 · Cited by 47,311 · see citing works
Lounging in a lysosome: the intracellular lifestyle of Coxiella burnetii
Voth, D. E., & Heinzen, R. A. (2007) · Cellular Microbiology · doi:10.1111/j.1462-5822.2007.00901.x · Cited by 11,773 · see citing works
Addiction Rare in Patients Treated with Narcotics
(1980) · New England Journal of Medicine · doi:10.1056/nejm198001103020221 · Cited by 1,106 · see citing works
Phan, K. L., Fitzgerald, D. A., Nathan, P. J., & Tancer, M. E. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.08.012 · Cited by 539 · see citing works
How citation distortions create unfounded authority: analysis of a citation network
Greenberg, S. A. (2009) · BMJ · doi:10.1136/bmj.b2680 · Cited by 499 · see citing works
A 1980 Letter on the Risk of Opioid Addiction
Leung, P. T. M., Macdonald, E. M., Stanbrook, M. B., Dhalla, I. A., & Juurlink, D. N. (2017) · New England Journal of Medicine · doi:10.1056/nejmc1700150 · Cited by 193 · see citing works
Mars, B., Heron, J., Biddle, L., Donovan, J., Holley, R., Piper, M., Potokar, J. P., Wyllie, C., & Gunnell, D. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.06.001 · Cited by 131 · see citing works
Scientific citations in Wikipedia
Nielsen, F. Å. (2007) · First Monday · doi:10.5210/fm.v12i8.1997 · Cited by 127 · see citing works
Examining Suicide-Risk Individuals Who Go Online for Suicide-Related Purposes
Harris, K. M., McLean, J. P., & Sheffield, J. K. (2009) · Archives of Suicide Research · doi:10.1080/13811110903044419 · Cited by 111 · see citing works
Suelzer, E. M., Deal, J., Hanus, K. L., Ruggeri, B., Sieracki, R., & Witkowski, E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.15552 · Cited by 69 · see citing works
Improving Wikipedia's credibility: References and citations in a sample of history articles
Luyt, B., & Tan, D. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21304 · Cited by 59 · see citing works
Offline Versus Online Suicide-Related Help Seeking: Changing Domains, Changing Paradigms
Seward, A.-L., & Harris, K. M. (2016) · Journal of Clinical Psychology · doi:10.1002/jclp.22282 · Cited by 56 · see citing works
Science through Wikipedia: A novel representation of open knowledge through co-citation networks
Arroyo‐Machado, W., Torres‐Salinas, D., Herrera‐Viedma, E., & Romero‐Frías, E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0228713 · Cited by 46 · see citing works
Altered time course of amygdala activation during speech anticipation in social anxiety disorder
Davies, C. D., Young, K., Torre, J. B., Burklund, L. J., Goldin, P. R., Brown, L. A., Niles, A. N., Lieberman, M. D., & Craske, M. G. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.11.014 · Cited by 21 · see citing works
Citations with identifiers in Wikipedia
Halfaker, A., Mansurov, B., Реди, М., et al. (2015) · Figshare · doi:10.6084/m9.figshare.1299540 · Cited by 14 · see citing works
Which Type of Research is Cited More Often in Wikipedia? A Case Study of PubMed Research
Dehdarirad, T., Didegah, F., & Sotudeh, H. (2018) · Chalmers Research (Chalmers University of Technology) · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [15]Association between Amygdala Hyperactivity to Harsh Faces and Severity of Social Anxiety in Generalized Social Phobia scite. 16256956
- [18]Examining Suicide-Risk Individuals Who Go Online for Suicide-Related Purposes scite. 19591000
Cited by
Sorted by citations
Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00146 · Cited by 160 · see citing works
“In another example from our own group, Nicholson et al. (2020) examined scientific papers cited in Wikipedia articles and found that — like the scientific literature as a whole — the vast majority presented findings that have not been subsequently verified.”
BackgroundSelf-citationBrody, S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.1331 · Cited by 24 · see citing works
Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021) · bioRxiv (published in Quantitative Science Studies) · doi:10.1101/2021.03.15.435418
Self-citation
- Authors
- Nicholson, J. M., Uppala, A., Sieber, M., Grabitz, P., Mordaunt, M., & Rife, S. C.
- Affiliations
- scite, Inc. Brooklyn NY USA
- Charité – Universitätsmedizin Berlin (DE)
- Murray State University (US)
- Corresponding author
- Nicholson, Joshua M.
- Venue
- The FEBS Journal
- Publisher
- Wiley
- Published
- 2020-11-19
- Volume
- 288
- Issue
- 14
- Pages
- 4242-4248
- ISSN
- 1742-464X (ISSN-L)
- 1742-4658
- PMC
- PMC8060352
- OpenAlex
- W3094504578
- Cited by
- 22
- References
- 18
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 5 · 2021: 4 · 2020: 1
- FWCI
- 2.28
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- FEBS Journal (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute on Drug Abuse (1R44DA050155‐01)
- U.S. Department of Health and Human Services
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Communication › Wikis in Education and Collaboration
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- Wikipedia, scientific citations, citation quality, supporting evidence, Web of Science, citation analysis
- MeSH terms
- Encyclopedias as Topic, Humans, Periodicals as Topic, Reproducibility of Results, Databases, Factual, Internet, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
