The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 201–300 of 1,003
Patel, P. R., Welle, E. J., Letner, J. G., Shen, H., Bullard, A. J., Caldwell, C. M., Vega-Medina, A., Richie, J. M., Thayer, H. E., Patil, P. G., et al. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/acab86 · Cited by 49
Fostering responsible research practices is a shared responsibility of multiple stakeholders
Bouter, L. M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.016 · Cited by 53
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 57
Out-of-the-Box Reproducibility: A Survey of Machine Learning Platforms
Isdahl, R., & Gundersen, O. E. (2019) · eScience · doi:10.1109/escience.2019.00017 · Cited by 80
Wyatt, T. D. (2020) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2019.0262 · Cited by 56
“Replications can fail to reproduce the results of the original study for a variety of reasons [4,10,46,48,90].”
BackgroundResultDuong‐Trung, N., Born, S., Kim, J. W., Schermeyer, M., Paulick, K., Borisyak, M., Bournazou, M. N. C., Werner, T., Scholz, R., Schmidt-Thieme, L., et al. (2022) · Biochemical Engineering Journal · doi:10.1016/j.bej.2022.108764 · Cited by 67
Preregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 52
Heunis, S., Lamerichs, R. M. J. N., Zinger, S., Caballero‐Gaudes, C., Jansen, J. F. A., Aldenkamp, A. P., & Breeuwer, M. M. (2020) · Human Brain Mapping · doi:10.1002/hbm.25010 · Cited by 57
“…1To prevent ambiguous interpretations of the more general terms “reproducibility,” “replicability,” and “repeatability,” in this work we adhere to the definitions for methods-, results-, and inferential reproducibility in the biomedical sciences proposed by Goodman, Fanelli, and Ioannidis (2016).”
MethodTowards computational reproducibility: researcher perspectives on the use and sharing of software
AlNoamany, Y., & Borghi, J. (2018) · PeerJ Computer Science · doi:10.7717/peerj-cs.163 · Cited by 55
“Attempts to provide a cohesive lexicon for describing reproducibility-related activities are described elsewhere ( Goodman, Fanelli & Ioannidis, 2016 ) but computational reproducibility generally refers to the description and sharing of software tools and data in such a manner as to enable their…”
BackgroundThe Circuit Motif as a Conceptual Tool for Multilevel Neuroscience
Braganza, O., & Beck, H. (2018) · Trends in Neurosciences · doi:10.1016/j.tins.2018.01.002 · Cited by 57
Pagell, M. D. (2020) · Journal of Operations Management · doi:10.1002/joom.1120 · Cited by 52
Bridgeford, E., Wang, S., Wang, Z., Xu, T., Craddock, R. C., Dey, J. K., Kiar, G., Gray-Roncal, W., Colantuoni, C., Douville, C. B., et al. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009279 · Cited by 53
“The concept of reproducibility is closely related to the statistical concepts of stability [5] and robustness [4].”
BackgroundThe Reproducibility Crisis and Academic Libraries
Sayre, F., & Riegelman, A. (2018) · College & Research Libraries · doi:10.5860/crl.79.1.2 · Cited by 47
Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47
Towards reproducible radiomics research: introduction of a database for radiomics studies
Akinci D’Antonoli, T., Cuocolo, R., Baessler, B., & Pinto dos Santos, D. (2023) · European Radiology · doi:10.1007/s00330-023-10095-3 · Cited by 49
“In the absence of a universally accepted definition of reproducibility, in this paper, we accept the earlier description of “research reproducibility” [13, 14].”
MethodKosmidou, C., Efstathiou, N. E., Hoang, M. V., Notomi, S., Konstantinou, E. K., Hirano, M., Takahashi, K., Maidana, D. E., Tsoka, P., YOUNG, L. H. Y., et al. (2018) · Scientific Reports · doi:10.1038/s41598-017-17634-1 · Cited by 47
A Comparison of DNA/RNA Extraction Protocols for High-Throughput Sequencing of Microbial Communities
Shaffer, J. P., Marotz, C. A., Belda‐Ferre, P., Martino, C., Wandro, S., Estaki, M., Salido, R. A., Carpenter, C. S., Zaramela, L. S., Minich, J. J., et al. (2021) · BioTechniques · doi:10.2144/btn-2020-0153 · Cited by 54
Mayo‐Wilson, E., Grant, S., Supplee, L. H., Kianersi, S., Amin, A., DeHaven, A., & Mellor, D. T. (2021) · Research Integrity and Peer Review · doi:10.1186/s41073-021-00112-8 · Cited by 57
“Evidence that much empirical research cannot be reproduced [2]—including basic experiments in psychology [3], economics [4], and the social sciences [5]—has led to concerns about responsible research conduct and a “reproducibility crisis” [6], with a consequent “credibility revolution” focused largely on increasing re…”
BackgroundPackaging data analytical work reproducibly using R (and friends)Preprint
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · doi:10.7287/peerj.preprints.3192v1 · Cited by 50
Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis
Nelson, N. C., Ichikawa, K., Chung, J., & Malik, M. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254090 · Cited by 51
Critical considerations of the contribution of the corticomotoneuronal pathway to central fatigue
Amann, M., Sidhu, S. K., McNeil, C. J., & Gandevia, S. C. (2022) · The Journal of Physiology · doi:10.1113/jp282564 · Cited by 57
“Finally, we draw attention to the frequently discussed, widespread problem with the replication and reproducibility of research findings (Ioannidis, 2005; Goodman et al., 2016).”
BackgroundIrreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy
Flier, J. S. (2016) · Molecular Metabolism · doi:10.1016/j.molmet.2016.11.006 · Cited by 45
“…of published bioscience research: Diagnosis, pathogenesis and therapy Jeffrey S. Flier was named the 21st Dean of the Faculty of Medicine at Harvard University on July 11, 2007, at which point he became the Caroline Shields Walker Professor of Medicine at Harvard Medical School.”
BackgroundMeyer, F., Bitsch, A., Forman, H. J., Fragoulis, A., Ghezzi, P., Henschenmacher, B., Kellner, R. L. L., Kühne, J., Ludwig, T., Sachno, D., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108940 · Cited by 59
Azad, T. D., Ehresman, J., Ahmed, A. K., Staartjes, V. E., Lubelski, D., Stienen, M. N., Veeravagu, A., & Ratliff, J. K. (2020) · The Spine Journal · doi:10.1016/j.spinee.2020.10.006 · Cited by 49
Strong Inference in Mathematical Modeling: A Method for Robust Science in the Twenty-First Century
Ganusov, V. V. (2016) · Frontiers in Microbiology · doi:10.3389/fmicb.2016.01131 · Cited by 47
“The definition of reproducibility may be difficult in general as it may vary by researcher (Goodman et al., 2016).”
BackgroundParola, A., Simonsen, A., Lin, J. M., Zhou, Y., Wang, H., Ubukata, S., Koelkebeck, K., Bliksted, V., & Fusaroli, R. (2022) · Schizophrenia Bulletin · doi:10.1093/schbul/sbac128 · Cited by 55
Jalali, M. S., DiGennaro, C., Guitar, A., Lew, K., & Rahmandad, H. (2021) · Epidemiologic Reviews · doi:10.1093/epirev/mxab006 · Cited by 45
“Reproducibility is the cornerstone of scientific research, and the case for enhancing reproducibility has been made repeatedly and has given rise to various guidelines and methodologies for authors to produce reproducible simulation models (25–30).”
Preclinical multi-target strategies for myocardial ischemia-reperfusion injury
Li, Y., Gao, Y. Q., & Li, G. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.967115 · Cited by 52
“Why preclinical experiments need rigor and reproducibility Multicenter tests that consider the robustness of interventions to numerous confounders and unknown variables are an exception in studies of cardioprotective strategies (164, 165).”
BackgroundReplications in agricultural economics
Finger, R., Grebitus, C., & Henningsen, A. (2023) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13386 · Cited by 56
Three Dimensions of Reproducibility in Natural Language Processing
Cohen, K. B., Xia, J., Zweigenbaum, P., Callahan, T. J., Hargraves, O., Goss, F. R., Ide, N., Névéol, A., Grouin, C., & Hunter, L. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/58wtisabky5x · Cited by 72
“…prevents us from operationalizing the solutions to the problems pointed out by writers on the topic of replicability and repeatability problems in natural language processing— we have no “clear operational criteria for what constitutes successful replication or reproduction” (Goodman et al., 2016).”
BackgroundMethodInfluentialEffect size and statistical power in the rodent fear conditioning literature – A systematic review
Carneiro, C. F. D., Moulin, T. C., Macleod, M. R., & Amaral, O. B. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196258 · Cited by 54
“This point is especially important to consider in the light of recent calls for basic science to use data synthesis tools such as meta-analysis (10) and formal or informal Bayesian inference (2,7,9,41).”
BackgroundMethodA taxonomy for reproducible and replicable research in environmental modelling
Essawy, B. T., Goodall, J. L., Voce, D., Morsy, M. M., Sadler, J. M., Choi, Y.-D., Tarboton, D. G., & Malik, T. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104753 · Cited by 50
“Several scientists have made efforts to define reproducibility (Easterbrook, 2014; Goodman et al., 2018; Gorgolewski and Poldrack, 2016; Ioannidis et al., 2009; Stodden et al., 2013).”
BackgroundFerland‐Beckham, C., Chaby, L. E., Daskalakis, N. P., Knox, D., Liberzon, I., Lim, M. M., McIntyre, C. K., Perrine, S. A., Risbrough, V. B., Sabban, E. L., et al. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.652636 · Cited by 53
“…they include such phenomena as a lack of scientific rigor, low statistical power, positive publication bias, a lack of preplanned statistical analyses, and poor or incomplete reporting of methodological detail in accordance with the ARRIVE 2.0 guidelines, among others (Goodman et al., 2016).”
InfluentialAntunes, B. A., & Hill, D. R. C. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100655 · Cited by 78
Opinion versus evidence for the need to move away from animal testing
Härtung, T. (2017) · ALTEX · doi:10.14573/altex.1703291 · Cited by 54
“Noteworthy, what is meant by “reproducibility” needs some sharpening here, as it means different things in different disciplines and areas (Goodman et al., 2016).”
BackgroundTierney, B. T., Tan, Y., Yang, Z., Shui, B., Walker, M. J., Kent, B. M., Kostic, A. D., & Patel, C. J. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001556 · Cited by 45
“It is for this reason that we hesitate to claim that we are holistically measuring reproducibility, as the requirements for rigorously doing so are nebulous [30] and would likely require considering all analytic biases that pervade microbiome analyses, something outside the scope of this project.”
BackgroundWhat is reproducibility in artificial intelligence and machine learning research?
Desai, A., Abdelhamid, M., & Padalkar, N. R. (2025) · AI Magazine · doi:10.1002/aaai.70004 · Cited by 42
Bergeat, D., Lombard, N., Gasmi, A., Le Floch, B., & Naudet, F. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.15209 · Cited by 41
A Framework for Improving the Quality of Research in the Biological Sciences
Casadevall, A., Ellis, L. M., Davies, E. W., McFall-Ngai, M., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.01256-16 · Cited by 46
“Colloquium participants agreed that there is a problem of reproducibility in the biological literature, although there was also a recognition that even the term “reproducibility” can be problematic since it can mean different things to different people (9).”
BackgroundKendler, K. S., Ohlsson, H., Fagan, A. A., Lichtenstein, P., Sundquist, J., & Sundquist, K. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2337 · Cited by 53
A step-by-step guide on preregistration and effective data sharing for psychopathology research.
Krypotos, A., Klugkist, I. G., Mertens, G., & Engelhard, I. M. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000424 · Cited by 44
Kaczkurkin, A. N., Moore, T. M., Sotiras, A., Xia, C. H., Shinohara, R. T., & Satterthwaite, T. D. (2019) · Biological Psychiatry · doi:10.1016/j.biopsych.2019.12.015 · Cited by 46
Applications of the microphysiology systems database for experimental ADME-Tox and disease models
Schurdak, M. E., Vernetti, L. A., Bergenthal, L., Wolter, Q. K., Shun, T. Y., Karcher, S. C., Taylor, D. L., & Gough, A. (2020) · Lab on a Chip · doi:10.1039/c9lc01047e · Cited by 42
The Evolution of Behavior Analysis: Toward a Replication Crisis?
Locey, M. L. (2020) · Perspectives on Behavior Science · doi:10.1007/s40614-020-00264-w · Cited by 47
Kumpfer, K. L., Scheier, L. M., & Brown, J. (2020) · Evaluation & the Health Professions · doi:10.1177/0163278718772886 · Cited by 41
“of the evidence for an intervention obtained from multiple studies (Goodman et al., 2016), and hopefully ones that are conducted by independent research teams.”
BackgroundClinicaDL: An open-source deep learning software for reproducible neuroimaging processing
Thibeau–Sutre, E., Díaz, M., Hassanaly, R., Routier, A., Dormont, D., Olivier Colliot, A., & Burgos, N. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106818 · Cited by 55
“Finally, Goodman et al. [21] and Stodden et al. [27] encourage others to report all the explored paths and negative results to be more transparent and to avoid potential bias in reporting.”
InfluentialFour principles for improved statistical ecology
Popovic, G. C., Mason, T. J., Drobniak, S. M., Marques, T. A., Potts, J., Joo, R., Altwegg, R., Burns, C. C. I., Mccarthy, M. A., Johnston, A., et al. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14270 · Cited by 64
“While definitions vary (Goodman et al., 2016 ), we will use the Turing Way Community ( 2023 ) defi - nition, which says a result is reproducible when the same analytical steps performed on the same dataset consistently produce the same answer.”
MethodStracke, C. M. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3_2 · Cited by 54
“However, I agree with Goodman et al. (2016) that it is still unclear how reproducibility relates and leads to the development of cumulative evidence and commonly accepted truth in the research community.”
BackgroundAquarium: open-source laboratory software for design, execution and data management
Vrana, J. D., de Lange, O., Yang, Y., Newman, G., Saleem, A., Miller, A., Cordray, C., Halabiya, S. F., Parks, M., Lopez, E., et al. (2021) · Synthetic Biology · doi:10.1093/synbio/ysab006 · Cited by 41
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 67
Role of Environment and Experimenter in Reproducibility of Behavioral Studies With Laboratory Mice
Nigri, M., Åhlgren, J., Wolfer, D. P., & Võikar, V. (2022) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2022.835444 · Cited by 46
Toward Open and Reproducible Epidemiology
Mathur, M. B., & Fox, M. P. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad007 · Cited by 55
“It is promising that some leaders in epidemiology (22, 23, 77) and journals are encouraging open science.”
BackgroundCo-simulation for buildings and smart energy systems — A taxonomic review
Alfalouji, Q., Schranz, T., Falay, B., Wilfling, S., Exenberger, J., Mattausch, T., Gomes, C. Â. G., & Schweiger, G. (2023) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2023.102770 · Cited by 47
Catalá-López, F., Caulley, L., Ridao-López, M., Hutton, B., Husereau, D., Drummond, M., Alonso‐Arroyo, A., Pardo-Fernández, M., Bernal‐Delgado, E., Meneu, R., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034463 · Cited by 41
Best practices for data management and sharing in experimental biomedical research
Cunha‐Oliveira, T., Ioannidis, J. P. A., & Oliveira, P. J. (2024) · Physiological Reviews · doi:10.1152/physrev.00043.2023 · Cited by 56
Self-citationManninen, T., Aćimović, J., Havela, R., Teppola, H., & Linne, M. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2018.00020 · Cited by 47
“However, there is an ongoing discussion on optimal use of terminology and several alternatives have been proposed (see e.g., Goodman et al., 2016; Rougier et al., 2017; Benureau and Rougier, 2018).”
BackgroundEnabling precision medicine via standard communication of HTS provenance, analysis, and results
Alterovitz, G., Dean, D. A., Goble, C., Crusoe, M. R., Soiland‐Reyes, S., Bell, A. L., Hayes, A., Suresh, A., Purkayastha, A., King, C. H., et al. (2018) · PLoS Biology · doi:10.1371/journal.pbio.3000099 · Cited by 41
“Researchers utilize two types of reproducibility: method reproducibility and result reproducibility[19].”
Method“Bad smells” in software analytics papers
Menzies, T., & Shepperd, M. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.04.005 · Cited by 41
“In addition, not all materials ( data and code) are available [41, 79].”
BackgroundTreble‐Barna, A., Heinsberg, L. W., Stec, Z., Breazeale, S., Davis, T. S., Kesbhat, A., Chattopadhyay, A., VonVille, H. M., Ketchum, A., Yeates, K. O., et al. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105078 · Cited by 39
Selective outcome reporting in randomized clinical trials of dental implants
Sendyk, D. I., Rovai, E. S., Souza, N. V., Deboni, M. C. Z., & Pannuti, C. M. (2019) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13128 · Cited by 35
The credibility crisis in research: Can economics tools help?
Gall, T., Ioannidis, J. P. A., & Maniadis, Z. (2017) · PLoS Biology · doi:10.1371/journal.pbio.2001846 · Cited by 35
Self-citationMarwick, B. (2017) · ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) · doi:10.17605/osf.io/3d6xx · Cited by 41
Effects of stimulus pulse rate on somatosensory adaptation in the human cortex
Hughes, C. L., Flesher, S. N., & Gaunt, R. A. (2022) · Brain stimulation · doi:10.1016/j.brs.2022.05.021 · Cited by 36
Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science
Murphy, J., Mesquida, C., Caldwell, A. R., Earp, B. D., & Warne, J. P. (2022) · Sports Medicine · doi:10.1007/s40279-022-01749-1 · Cited by 46
“probability that a claim is robust based on the combination of the new study evidence with the plausible prior distribution of the data [50].”
BackgroundA practical guide for transparency in psychological sciencePreprint
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H. R., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/rtygm · Cited by 46
What is Reproducibility in Artificial Intelligence and Machine Learning Research?
Desai, A., Abdelhamid, M., & Padalkar, N. R. (2024) · arXiv · doi:10.48550/arxiv.2407.10239 · Cited by 36
Rethinking research reproducibility
Dirnagl, U. (2019) · The EMBO Journal · doi:10.15252/embj.2018101117 · Cited by 35
“A conceptual framework for “reproducibil-ity” was laid out by Goodman et al (2016), which discriminates between three basic concepts: (i) Methods reproducibility, which requires that procedures of a study must be described in such detail that an expert can faithfully repeat it.”
Open Science Needs No Martyrs, but We Must Recognize the Need for Reform
Susi, T. (2022) · doi:10.4000/books.oep.16156 · Cited by 36
How measurement science can improve confidence in research results
Plant, A. L., Becker, C. A., Hanisch, R. J., Boisvert, R. F., Possolo, A. M., & Elliott, J. T. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2004299 · Cited by 35
Radical Solutions and Open Science: An Open Approach to Boost Higher Education
Burgos, D. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3 · Cited by 51
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
Tao, M., Xiang, Y., Zhao, D., Shan, P., & Yang, Y. (2022) · Communications Materials · doi:10.1038/s43246-022-00273-z · Cited by 35
An Action Plan for Diamond Open Access
Borrell-Damián, L., & Rooryck, J. (2022) · doi:10.4000/books.oep.16147 · Cited by 60
Why do probabilistic clinical models fail to transport between sites
Lasko, T. A., Strobl, E. V., & Stead, W. W. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01037-4 · Cited by 61
Cueno, M. E., & Ochiai, K. (2018) · Frontiers in Immunology · doi:10.3389/fimmu.2018.01158 · Cited by 34
A framework for clinical and translational research in the era of rigor and reproducibility
Wichman, C., Smith, L. M., & Yu, F. (2020) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.523 · Cited by 43
Denaxas, S., Direk, K., González-Izquierdo, A., Pikoula, M., Cakiroglu, S. A., Moore, J. H., Hemingway, H., & Smeeth, L. L. (2017) · BioData Mining · doi:10.1186/s13040-017-0151-7 · Cited by 35
“Semantically similarly concepts such as “replicability” and “duplication” are often used interchangeably [20].”
BackgroundBasson, A. R., LaSalla, A., Lam, G., Kulpins, D., Moen, E. L., Sundrud, M. S., Miyoshi, J., Ilić, S., Theriault, B. R., Cominelli, F., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-60900-y · Cited by 35
Placidi, L., Lenkowicz, J., Cusumano, D., Boldrini, L., Dinapoli, N., & Valentini, V. (2020) · Physica Medica · doi:10.1016/j.ejmp.2020.07.022 · Cited by 33
Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?
Iso‐Ahola, S. E. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00879 · Cited by 41
“It has been suggested that reproducibility is also unattainable because of problems and variations in “methods,” “results,” and “inferences” regarding replications of the original findings (Goodman et al., 2016).”
ResultInfluentialIntramitochondrial proteostasis is directly coupled to α-synuclein and amyloid β1-42 pathologies
Lautenschläger, J., Wagner-Valladolid, S., Stephens, A. D., Fernández‐Villegas, A., Hockings, C., Mishra, A., Manton, J. D., Fantham, M., Lü, M., Rees, E. J., et al. (2020) · Journal of Biological Chemistry · doi:10.1074/jbc.ra119.011650 · Cited by 39
Jaric, I., Voelkl, B., Clerc, M., Schmid, M. W., Novak, J., Rosso, M., Rufener, R., von Kortzfleisch, V. T., Richter, S. H., Buettner, M., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001837 · Cited by 32
Müller-Hansen, F., Callaghan, M. W., & Minx, J. C. (2020) · Energy Research & Social Science · doi:10.1016/j.erss.2020.101691 · Cited by 32
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32
Wagner, S. J., Matek, C., Boushehri, S. S., Boxberg, M., Lamm, L., Sadafi, A., Waibel, D., Marr, C., & Peng, T. (2023) · Modern Pathology · doi:10.1016/j.modpat.2023.100350 · Cited by 38
Exact replication: Foundation of science or game of chance?
Piper, S. K., Grittner, U., Rex, A., Riedel, N., Fischer, F. H., Nadon, R., Siegerink, B., & Dirnagl, U. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000188 · Cited by 36
Creating a neuroprosthesis for active tactile exploration of textures
O’Doherty, J. E., Shokur, S., Medina, L. E., Lebedev, M., & Nicolelis, M. A. L. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1908008116 · Cited by 36
Qresp, a tool for curating, discovering and exploring reproducible scientific papers
Govoni, M., Munakami, M., Tanikanti, A., Skone, J. H., Runesha, H. B., Giberti, F., Pablo, J., & Galli, G. (2019) · Scientific Data · doi:10.1038/sdata.2019.2 · Cited by 34
Moassefi, M., Rouzrokh, P., Conte, G. M., Vahdati, S., Fu, T., Tahmasebi, A., Younis, M., Farahani, K., Gentili, A., Kline, T. L., et al. (2023) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-023-00870-5 · Cited by 43
“Technical reproducibility refers to the ability to reproduce a paper’s results precisely as presented in the paper under the same condition [10, 11].”
BackgroundWriting Empirical Articles: Transparency, Reproducibility, Clarity, and Memorability
Gernsbacher, M. A. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918754485 · Cited by 47
Independent discussion sections for improving inferential reproducibility in published research
Avidan, M. S., Ioannidis, J. P. A., & Mashour, G. A. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2018.12.010 · Cited by 34
Self-citationAMRcloud: a new paradigm in monitoring of antibiotic resistance
Kuzmenkov, A. Y., Vinogradova, A. G., Trushin, I. V., Avramenko, A. A., Edelstein, M. V., Dekhnich, A. V., & Kozlov, R. S. (2019) · Clinical Microbiology and Antimicrobial Chemotherapy · doi:10.36488/cmac.2019.2.119-124 · Cited by 35
“Организованная и качественно оформленная работа позволяет упростить процесс повторного анализа, не только подтверждая результаты, но и его прозрачность с последующим рассмотрением с точки зрения компьютерных наук [14].”
Witjes, V. M., Boleij, A., & Halffman, W. (2020) · Animals · doi:10.3390/ani10122415 · Cited by 39
“Thirdly, results reproducibility, which involves a new independent experiment (also referred to as a replication study) verifying the results and data [19].”
BackgroundMethodMansmann, U., Locher, C., Praßer, F., Weissgerber, T. L., Sax, U., Posch, M., Decullier, É., Cristea, I. A., Debray, T. P. A., Held, L., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02080-y · Cited by 32
Self-citationImplementing code review in the scientific workflow: Insights from ecology and evolutionary biology
Ivimey‐Cook, E. R., Pick, J. L., Bairos‐Novak, K. R., Čulina, A., Gould, E., Grainger, M., Marshall, B. M., Moreau, D., Paquet, M., Royauté, R., et al. (2023) · Journal of Evolutionary Biology · doi:10.1111/jeb.14230 · Cited by 42
“The last ‘R’ of code review builds on the previous code review stages, and is perhaps the most fundamental: can the code produce the output, and thus support the conclusions, given in the paper (Goodman et al., 2016; Figure 1 , Box 1 in Data S1)?”
ResultA theory and methodology to quantify knowledge
Fanelli, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181055 · Cited by 44
“measured and understood in different fields (see [32,62]).”
BackgroundResultSelf-citationWhat the replication crisis means for intervention science
Hillary, F. G., & Medaglia, J. D. (2019) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2019.05.006 · Cited by 36
“, 2015), there is growing consensus that there is room for improvement in approach and methods used and these concerns have spared few areas of research in the health sciences (Benjamin et al., 2018; Friesike et al., 2015; Goodman et al., 2016; Munafò et al., 2017; Nosek et al., 2015b).”
MethodRandomness In Neural Network Training: Characterizing The Impact of ToolingPreprint
Zhuang, D., Zhang, X., Song, S. L., & Hooker, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11872 · Cited by 78
“Reproducibility in machine learning As numerous works have pointed out [Goodman et al., 2016, Gunder-sen and Kjensmo, 2018, Barba, 2018, Drummond, 2009], the word reproducibility can correspond to very different standards, ranging from the ability to reproduce statistically similar values [Raff,…”
BackgroundGenomic reproducibility in the bioinformatics era
Baykal, P. I., Łabaj, P. P., Markowetz, F. M., Schriml, L. M., Stekhoven, D. J., Mangul, S., & Beerenwinkel, N. (2024) · Genome biology · doi:10.1186/s13059-024-03343-2 · Cited by 46
“[2] as results reproducibility and by Gundersen [4] as outcome reproducibility.”
MethodResultInfluentialWeller, D. L., Kovač, J., Kent, D. J., Roof, S. E., Tokman, J. I., Mudrak, E. L., Kowalcyk, B. B., Oryang, D., de Aceituno, A. M. F., & Wiedmann, M. (2017) · Food Microbiology · doi:10.1016/j.fm.2017.06.009 · Cited by 30
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.