Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.

Prediction of robust scientific facts from literature

Open access
Belikov, Alexander V., Rzhetsky, Andrey, Evans, James · 2022 · Nature Machine Intelligence

Abstract

The growth of published science in recent years has escalated the difficulty that human and algorithmic agents face in reasoning over prior knowledge to select the next experiment. This challenge is increased by uncertainty about the reproducibility of published findings. The availability of massive digital archives, machine reading, extraction tools and automated high-throughput experiments allows us to evaluate these challenges computationally at scale and identify novel opportunities to craft policies that accelerate scientific progress. Here we demonstrate a Bayesian calculus that enables positive prediction of robust scientific claims with findings extracted from published literature, weighted by scientific, social and institutional factors demonstrated to increase replicability. Illustrated with the case of gene regulatory interactions, our approach automatically estimates and counteracts sources of bias, revealing that scientifically focused but socially and institutionally diverse research activity is most likely to replicate. This results in updated certainty about the literature, which accurately predicts robust scientific facts on which new experiments should build. Our findings allow us to identify and evaluate policy recommendations for scientific institutions that may increase robust scientific knowledge, including sponsorship of increased diversity of and independence between investigations of any particular scientific phenomenon, and diversity of scientific ph…

Overview

Integrity signals

Registry

Crossref · 10.1038/s42256-022-00474-8

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 56 references

Research works printed but not linked: 6

  1. Foreman, P. Weimar culture, causality and quantum theory 1918–1927. Hist. Stud. Phys. Biol. Sci. 3, 2–225 (1971).
  2. Borenstein, M., Hedges, L. V., Higgins, J. P. T. & Rothstein, H. R. Introduction to Meta-Analysis (Wiley, 2011).
  3. Nussbaum, D. The role of conceptual replication. Psychologist 25, 350 (2012).
  4. Scargle, J. D. Publication bias (the ‘file-drawer problem’) in scientific inference. Preprint at https://arxiv.org/abs/physics/9909033 (1999). · arXiv
  5. Sunstein, C. R. Republic.com (Princeton Univ. Press, 2001).
  6. Rzhetsky, A. et al. GeneWays: a system for extracting, analyzing, visualizing, and integrating molecular pathway data. J. Biomed. Inform. 37, 43–53 (2004). · DOI · Paper page

Cited by

See all 36

Metadata

Authors
Belikov, A. V., Rzhetsky, A., & Evans, J.
Affiliations
  • University of Chicago (US)
  • Santa Fe Institute (US)
Corresponding author
  • Belikov, Alexander V.
  • Evans, James
Venue
Nature Machine Intelligence
Published
2022-04-28
Volume
4
Issue
5
Pages
445-454
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4293240672
Cited by
30
References
56
Citations by year
2026: 4 · 2025: 12 · 2024: 6 · 2023: 6 · 2022: 2
FWCI
3.93
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • United States Department of Defense | Defense Advanced Research Projects Agency (14145043, HR00111820006)
  • United States Department of Defense | United States Air Force | AFMC | Air Force Office of Scientific Research (FA9550-15-1-0162)
  • National Science Foundation (1158803)
  • John Templeton Foundation (Metaknowledge Network)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
Keywords
replicability, Bayesian forecasting, scientific literature mining, gene regulatory interactions, diversity in research, research autonomy, bias correction, science policy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Prediction of robust scientific facts from literature. https://app.cochat.ai/papers/10.1038/s42256-022-00474-8
The paper
Belikov, A. V., Rzhetsky, A., & Evans, J. (2022). Prediction of robust scientific facts from literature. Nature Machine Intelligence. https://doi.org/10.1038/s42256-022-00474-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.