Reproducibility and Replicability in Science
Abstract
One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI's tools and processes challenge core norms and values in the conduct of science, including accountability, transparency, replicability, and human responsibility.
“its tools and processes challenge core norms and values in the conduct of science, including accountability, transparency, replicability, and human responsibility (1–3).”
From Protecting scientific integrity in an age of generative AI (2024) · cites this paper as [2]
Cited by
Sorted by citations
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410 · see citing works
Gopalakrishna, G., ter Riet, G., Vink, G., Stoop, I., Wicherts, J. M., & Bouter, L. M. (2022) · PLOS ONE · doi:10.1371/journal.pone.0263023 · Cited by 296 · see citing works
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246 · see citing works
“The tools in this category allow users to formulate a natural language query and generate a list of relevant research papers sourced from online repositories.”
Replication across space and time must be weak in the social and environmental sciences
Goodchild, M. F., & Li, W. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2015759118 · Cited by 157 · see citing works
Patterns of Reliability: Assessing the Reproducibility and Integrity of DNA Methylation Measurement
Sugden, K., Hannon, E. J., Arseneault, L., Belsky, D. W., Corcoran, D. L., Fisher, H. L., Houts, R. M., Kandaswamy, R., Moffitt, T. E., Poulton, R., et al. (2020) · Patterns · doi:10.1016/j.patter.2020.100014 · Cited by 151 · see citing works
Assessment of transparency indicators across the biomedical literature: How open is open?
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001107 · Cited by 139 · see citing works
van IJzendoorn, M. H., Schuengel, C., Wang, Q., & Bakermans-Kranenburg, M. J. (2023) · Development and Psychopathology · doi:10.1017/s0954579421001462 · Cited by 116 · see citing works
Reproducibility and replicability: opportunities and challenges for geospatial research
Kedron, P., Li, W., Fotheringham, S., & Goodchild, M. (2021) · International Journal of Geographical Information Science · doi:10.1080/13658816.2020.1802032 · Cited by 108 · see citing works
Protecting scientific integrity in an age of generative AI
Blau, W., Cerf, V. G., Enriquez, J., Francisco, J. S., Gasser, U., Gray, M. L., Greaves, M., Grosz, B. J., Jamieson, K. H., Haug, G. H., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2407886121 · Cited by 86 · see citing works
Macleod, M., Collings, A. M., Graf, C., Kiermer, V., Mellor, D., Swaminathan, S., Sweet, D., & Vinson, V. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103238118 · Cited by 77 · see citing works
Where Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81 · see citing works
Harrison, T. M. (2020) · doi:10.1007/978-3-030-46687-9 · Cited by 53 · see citing works
Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., et al. (2023) · Psychological Science · doi:10.1177/09567976221140828 · Cited by 54 · see citing works
Dudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28 · see citing works
Dental Research Data Availability and Quality According to the FAIR Principles
Uribe, S. E., Sofi-Mahmudi, A., Raittio, E., Maldupa, I., & Vilne, B. (2022) · Journal of Dental Research · doi:10.1177/00220345221101321 · Cited by 27 · see citing works
Navarro Leija, O. S., Shiptoski, K., Scott, R. G., Wang, B., Renner, N., Newton, R. R., & Devietti, J. (2020) · Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3373376.3378519 · Cited by 22 · see citing works
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
“Our analysis is based on a widely accepted definition of reproducibility: that the code and data accompanying a paper should be enough to reproduce the results that it reports [37].”
Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11363 · Cited by 4 · see citing works
- Authors
- Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, Committee on Science, E., Affairs, P. A. G., & National Academies of Sciences, E.
- Venue
- National Academies Press eBooks
- Publisher
- National Academies Press
- Published
- 2019-09-20
- OpenAlex
- W2944196828
- Cited by
- 924
- References
- 0
- Citations by year
- 2026: 56 · 2025: 106 · 2024: 187 · 2023: 146 · 2022: 139 · 2021: 131 · 2020: 123 · 2019: 34 · 2018: 2
- FWCI
- 161.97
- Type
- book
- Language
- en
- Open access
- closed
- Locations
- National Academies Press eBooks (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reproducibility, replicability, research rigor, scientific discovery, research transparency, scholarly publishing, research misconduct
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
