The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Prediction of robust scientific facts from literature
Open accessAbstract
The growth of published science in recent years has escalated the difficulty that human and algorithmic agents face in reasoning over prior knowledge to select the next experiment. This challenge is increased by uncertainty about the reproducibility of published findings. The availability of massive digital archives, machine reading, extraction tools and automated high-throughput experiments allows us to evaluate these challenges computationally at scale and identify novel opportunities to craft policies that accelerate scientific progress. Here we demonstrate a Bayesian calculus that enables positive prediction of robust scientific claims with findings extracted from published literature, weighted by scientific, social and institutional factors demonstrated to increase replicability. Illustrated with the case of gene regulatory interactions, our approach automatically estimates and counteracts sources of bias, revealing that scientifically focused but socially and institutionally diverse research activity is most likely to replicate. This results in updated certainty about the literature, which accurately predicts robust scientific facts on which new experiments should build. Our findings allow us to identify and evaluate policy recommendations for scientific institutions that may increase robust scientific knowledge, including sponsorship of increased diversity of and independence between investigations of any particular scientific phenomenon, and diversity of scientific ph…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Both “machine scientists” (96, 97) and large language models hold the promise to help encapsulate the knowledge in the scientific literature for the use of scientists and the lay public.”
From The entities enabling scientific fraud at scale are large, resilient, and growing rapidly (2025) · cites this paper as [96]
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The contrast between published findings and experimental data suggests the extent of the ‘file drawer problem’ in science where scientists euphemistically ‘file’ but do not publish negative or inconclusive results 55,56 .”
ResultEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 2×
“These forces have led to widespread concerns regarding the reliability and reproducibility of findings in fields ranging from pharmacology 5,29–31 and genetics 3,4,11,32–34 to psychology 35,36 with widespread implications for the accumulation of certainty in science.”
BackgroundThe Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720
Maps of random walks on complex networks reveal community structure
Rosvall, M., & Bergstrom, C. T. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0706851105 · Cited by 4,583
The Increasing Dominance of Teams in Production of Knowledge
Wuchty, S., Jones, B. F., & Uzzi, B. (2007) · Science · doi:10.1126/science.1136099 · Cited by 3,615
A Next Generation Connectivity Map: L1000 Platform and the First 1,000,000 Profiles
Subramanian, A., Narayan, R., Corsello, S. M., Peck, D. D., Natoli, T. E., Lu, X., Gould, J., Davis, J. F., Tubelli, A. A., Asiedu, J. K., et al. (2017) · Cell · doi:10.1016/j.cell.2017.10.049 · Cited by 3,515
Groups of diverse problem solvers can outperform groups of high-ability problem solvers
Hong, L., & Page, S. E. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0403723101 · Cited by 2,103
Lohmueller, K. E., Pearce, C. L., Pike, M., Lander, E. S., & Hirschhorn, J. N. (2003) · Nature Genetics · doi:10.1038/ng1071 · Cited by 1,900
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Research works printed but not linked: 6
- Foreman, P. Weimar culture, causality and quantum theory 1918–1927. Hist. Stud. Phys. Biol. Sci. 3, 2–225 (1971).
- Borenstein, M., Hedges, L. V., Higgins, J. P. T. & Rothstein, H. R. Introduction to Meta-Analysis (Wiley, 2011).
- Nussbaum, D. The role of conceptual replication. Psychologist 25, 350 (2012).
- Scargle, J. D. Publication bias (the ‘file-drawer problem’) in scientific inference. Preprint at https://arxiv.org/abs/physics/9909033 (1999). · arXiv
- Sunstein, C. R. Republic.com (Princeton Univ. Press, 2001).
- Rzhetsky, A. et al. GeneWays: a system for extracting, analyzing, visualizing, and integrating molecular pathway data. J. Biomed. Inform. 37, 43–53 (2004). · DOI · Paper page
Cited by
Recognizing and utilizing novel research opportunities with artificial intelligence
von Krogh, G., Roberson, Q., & Gruber, M. (2023) · Academy of Management Journal · doi:10.5465/amj.2023.4002 · Cited by 164
Accelerating science with human-aware artificial intelligence
Sourati, J., & Evans, J. A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01648-z · Cited by 155
Self-citationThe entities enabling scientific fraud at scale are large, resilient, and growing rapidly
Richardson, R. A. K., Hong, S. S., Byrne, J. A., Stoeger, T., & Amaral, L. A. N. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2420092122 · Cited by 129
Zhang, Y., Sui, X., Pan, F., Yu, K., Li, K., Tian, S., Erdengasileng, A., Han, Q., Wang, W., Wang, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01014-w · Cited by 101
Quantifying the use and potential benefits of artificial intelligence in scientific research
Gao, J., & Wang, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02020-5 · Cited by 81
Metadata
- Authors
- Belikov, A. V., Rzhetsky, A., & Evans, J.
- Affiliations
- University of Chicago (US)
- Santa Fe Institute (US)
- Corresponding author
- Belikov, Alexander V.
- Evans, James
- Venue
- Nature Machine Intelligence
- Published
- 2022-04-28
- Volume
- 4
- Issue
- 5
- Pages
- 445-454
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4293240672
- Cited by
- 30
- References
- 56
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 6 · 2023: 6 · 2022: 2
- FWCI
- 3.93
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Nature Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- United States Department of Defense | Defense Advanced Research Projects Agency (14145043, HR00111820006)
- United States Department of Defense | United States Air Force | AFMC | Air Force Office of Scientific Research (FA9550-15-1-0162)
- National Science Foundation (1158803)
- John Templeton Foundation (Metaknowledge Network)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- replicability, Bayesian forecasting, scientific literature mining, gene regulatory interactions, diversity in research, research autonomy, bias correction, science policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Prediction of robust scientific facts from literature. https://app.cochat.ai/papers/10.1038/s42256-022-00474-8
- The paper
- Belikov, A. V., Rzhetsky, A., & Evans, J. (2022). Prediction of robust scientific facts from literature. Nature Machine Intelligence. https://doi.org/10.1038/s42256-022-00474-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.