The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 101–200 of 1,003
Reproducibility and Replicability in Geographical Analysis
Kedron, P., Frazier, A. E., Trgovac, A. B., NELSON, T. A., & Fotheringham, A. S. (2019) · Geographical Analysis · doi:10.1111/gean.12221 · Cited by 113
On the Role of Theory and Modeling in Neuroscience
Levenstein, D., Alvarez, V. A., Amarasingham, A., Azab, H., Chen, Z. S., Gerkin, R. C., Hasenstaub, A. R., Iyer, R., Jolivet, R., Marzen, S., et al. (2023) · Journal of Neuroscience · doi:10.1523/jneurosci.1179-22.2022 · Cited by 119
On the role of theory and modeling in neuroscience
Levenstein, D., Alvarez, V. A., Amarasingham, A., Azab, H., Chen, Z. S., Gerkin, R. C., Hasenstaub, A., Iyer, R., Jolivet, R. B., Marzen, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.13825 · Cited by 93
The fundamental principles of reproducibility
Gundersen, O. E. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0210 · Cited by 124
“can reveal different information that allows for different interpretation, which means that different investigators might interpret the analysis in different ways and thus reach different conclusions [34]. So, to conclude, our beliefs in scientific theories and hypotheses are strengthened with new evidence. Reproducing…”
BackgroundMethodResultInfluentialTraceability for Trustworthy AI: A Review of Models and Tools
Mora‐Cantallops, M., Sánchez‐Alonso, S., García‐Barriocanal, E., & Sicilia, M. (2021) · Big Data and Cognitive Computing · doi:10.3390/bdcc5020020 · Cited by 115
Signaling the trustworthiness of science
Jamieson, K., McNutt, M., Kiermer, V., & Sever, R. (2019) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1913039116 · Cited by 70
A statistical definition for reproducibility and replicabilityPreprint
Patil, P., Peng, R. D., & Leek, J. T. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/066803 · Cited by 106
Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
Semmelrock, H., Ross-Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14325 · Cited by 88
Laraway, S., Snycerski, S., Pradhan, S., & Huitema, B. E. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00193-3 · Cited by 103
“This should improve the SCV and inferential reproducibility of behavioral studies (Goodman et al., 2016).”
BackgroundThe immuneML ecosystem for machine learning analysis of adaptive immune receptor repertoires
Pavlović, M., Scheffer, L., Motwani, K., Kanduri, C., Kompova, R., Vazov, N., Waagan, K., Bernal, F. L. M., da Costa, A. A., Corrie, B., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00413-z · Cited by 94
“Note that here we are referring to reproducibility in the sense of repeating a computational analysis in its exact form, also referred to as methods reproducibility 69 , although the YAML files are also”
MethodReproducibility of clinical research in critical care: a scoping review
Niven, D. J., McCormick, T. J., Straus, S. E., Hemmelgarn, B. R., Jeffs, L. P., Barnes, T., & Stelfox, H. T. (2018) · BMC Medicine · doi:10.1186/s12916-018-1018-6 · Cited by 91
“Accepting the limitations of this approach [11], it is congruent with that employed in previous reproducibility research [14–17], and resulted in a rate of reproducible research that compares favorably with much of the existing clinical literature [4, 14–16].”
MethodResultInfluentialManiadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 105
“Goodman et al. (2016) solidify the existing discussion by proposing a general terminology to refer to key relevant concepts such as different types of reproducibility.”
Statistical considerations in reporting cardiovascular research
Lindsey, M. L., Gray, G. A., Wood, S. K., & Curran‐Everett, D. (2018) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00309.2018 · Cited by 87
“and reproducibility,(66, 135) the goals of guidelines are to provide best practice information 55”
BackgroundAssociation of Routine Preoperative Frailty Assessment With 1-Year Postoperative Mortality
Varley, P. R., Buchanan, D., Bilderback, A. L., Wisniewski, M. K., Johanning, J. M., Nelson, J. B., Johnson, J. T., Minnier, T. E., & Hall, D. E. (2023) · JAMA Surgery · doi:10.1001/jamasurg.2022.8341 · Cited by 94
A systematic review on food recommender systems
Bondevik, J. N., Bennin, K. E., Babur, Ö., & Ersch, C. (2023) · Expert systems with applications · doi:10.1016/j.eswa.2023.122166 · Cited by 87
“There is an increase in concerns regarding this issue in several research domains (Goodman, Fanelli, & Ioannidis, 2016; Ivie & Thain, 2018) such as drug development (Begley, 2013), psychology (Collaboration, 2015) and software engineering (Lewowski & Madeyski, 2022).”
BackgroundEnvironment, susceptibility windows, development, and child health
Wright, R. O. (2017) · Current Opinion in Pediatrics · doi:10.1097/mop.0000000000000465 · Cited by 94
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83
Investigating resting-state functional connectivity in the cervical spinal cord at 3 T
Eippert, F., Kong, Y., Winkler, A. M., Andersson, J., Finsterbusch, J., Büchel, C., Brooks, J. C. W., & Tracey, I. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.12.072 · Cited by 83
“Finally, we need to show that the results we obtain are robust against variations in the data analysis pipeline, thus ensuring their inferential reproducibility (Goodman et al., 2016).”
BackgroundMethodThe five pillars of computational reproducibility: bioinformatics and beyond
Ziemann, M., Poulain, P., & Bora, A. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad375 · Cited by 121
High Throughput Multi-Omics Approaches for Clinical Trial Evaluation and Drug Discovery
Zielinski, J., Luke, J. J., Guglietta, S., & Krieg, C. (2021) · Frontiers in Immunology · doi:10.3389/fimmu.2021.590742 · Cited by 87
Information Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research
Ioannidis, J. P. A. (2017) · JAMA · doi:10.1001/jama.2017.0549 · Cited by 78
Self-citationHarris, B. N. (2019) · General and Comparative Endocrinology · doi:10.1016/j.ygcen.2019.113355 · Cited by 83
Open Source, Open Science, and the Replication Crisis in HCI
Echtler, F., & Häussler, M. (2018) · CHI Extended Abstracts · doi:10.1145/3170427.3188395 · Cited by 85
“There is even some disagreement about what the terms "reproducibility" or "replicability" by themselves mean [9].”
BackgroundMagni, P., Bier, D. M., Pecorelli, S., Agostoni, C. V., Astrup, A. V., Brighenti, F., Cook, R., Folco, E., Fontana, L., Gibson, R. A., et al. (2017) · Advances in Nutrition · doi:10.3945/an.116.014738 · Cited by 97
“There are various aspects of reproducible research, ranging from the ability to recompute data analytic results given an observed dataset and knowledge of the pipeline (41) to reproducibility across different datasets (reproducibility of results) and reproducibility of inferences from the same datasets and analyses (4…”
Self-citationManterola, C., Grande, L., Otzen, T., García, N., Salazar, P., & Quiroz, G. (2018) · Revista chilena de infectología · doi:10.4067/s0716-10182018000600680 · Cited by 86
Poly(ADP-ribose) polymerase 1 searches DNA via a ‘monkey bar’ mechanism
Rudolph, J., Mahadevan, J., Dyer, P. N., & Luger, K. (2018) · eLife · doi:10.7554/elife.37818 · Cited by 75
Slow Waves Promote Sleep-Dependent Plasticity and Functional Recovery after Stroke
Facchin, L., Schöne, C., Mensen, A., Bandarabadi, M., Pilotto, F., Saxena, S., Libourel, P., Bassetti, C. L., & Adamantidis, A. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.0373-20.2020 · Cited by 84
Frequency Shapes the Quality of Tactile Percepts Evoked through Electrical Stimulation of the Nerves
Graczyk, E. L., Christie, B. P., He, Q., Tyler, D. J., & Bensmaïa, S. J. (2022) · Journal of Neuroscience · doi:10.1523/jneurosci.1494-21.2021 · Cited by 86
Analytical Nanoscience and Nanotechnology: Where we are and where we are heading
Soriano, M. L., Zougagh, M., Valcárcel, M., & Ríos, Á. (2017) · Talanta · doi:10.1016/j.talanta.2017.09.012 · Cited by 76
Konkol, M., Kray, C., & Pfeiffer, M. (2018) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2018.1508687 · Cited by 91
“Goodman et al .”
The War against Tuberculosis: A Review of Natural Compounds and Their Derivatives
Maiolini, M., Gause, S., Taylor, J., Steakin, T., Shipp, G. M., Lamichhane, P., Deshmukh, B., Shinde, V. M., Bishayee, A., & Deshmukh, R. R. (2020) · Molecules · doi:10.3390/molecules25133011 · Cited by 85
Reproducibility failures are essential to scientific inquiry
Redish, A. D., Kummerfeld, E., Morris, R. L., & Love, A. C. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806370115 · Cited by 66
Usui, T., Macleod, M. R., McCann, S., Senior, A. M., & Nakagawa, S. (2021) · PLoS Biology · doi:10.1371/journal.pbio.3001009 · Cited by 76
“We further discuss these results below in the context of their implications for improving the replicability (also defined as “Results reproducibility” [1]) and generalizability of preclinical studies.”
BackgroundRavi, N. S., Wienert, B., Wyman, S. K., Bell, H. W., George, A., Mahalingam, G., Vu, J. T., Prasad, K. M., Bandlamudi, B. P., Devaraju, N., et al. (2022) · eLife · doi:10.7554/elife.65421 · Cited by 77
Furniture for Quantitative Scientists
Tyson, S. B., & Brignone, E. (2017) · The R Journal · doi:10.32614/rj-2017-037 · Cited by 70
“1We use the definitions from Goodman et al. (2016): "methods reproducibility" refers to the ability to obtain the same results with the same methods and data and "results reproducibility" means being able to obtain similar results using the same methods with independent data.”
The SONATA data format for efficient description of large-scale network models
Dai, K., Hernando, J., Billeh, Y. N., Gratiy, S. L., Planas, J., Davison, A. P., Durá-Bernal, S., Gleeson, P., Devresse, A., Dichter, B., et al. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007696 · Cited by 93
“These challenges contribute to difficulties in closing the virtuous experiment/ modeling loop and to the overall reproducibility crisis[37,38,12].”
BackgroundDo machine learning platforms provide out-of-the-box reproducibility?
Gundersen, O. E., Shamsaliei, S., & Isdahl, R. J. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.06.014 · Cited by 84
A critical moment in machine learning in medicine: on reproducible and interpretable learning
Ciobanu-Caraus, O., Aicher, A., Kernbach, J. M., Regli, L., Serra, C., & Staartjes, V. E. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-05892-8 · Cited by 97
“Reproducibility refers to the ability of an independent research group to reproduce the results of an original study using the same data and code [23, 47].”
Reproducible preclinical research—Is embracing variability the answer?
Karp, N. A. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2005413 · Cited by 77
Schoemann, M., O’Hora, D. P., Dale, R. A. C., & Scherbaum, S. (2020) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01851-3 · Cited by 76
Understanding experiments and research practices for reproducibility: an exploratory study
Samuel, S., & König‐Ries, B. (2021) · PeerJ · doi:10.7717/peerj.11140 · Cited by 76
“Scientists have provided different definitions of the term reproducibility ( Taylor & Kuyatt, 1994 ; Goodman, Fanelli & Ioannidis, 2016 ; ACM, 2017 ; Plesser, 2018 ; ACM, 2020 ) and a standard definition is still not agreed upon ( Baker, 2016b ).”
BackgroundMcDermott, R. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101675 · Cited by 122
Which Melbourne? Augmenting Geocoding with Maps
Gritta, M., Pilehvar, M. T., & Collier, N. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1119 · Cited by 104
“We made all of our resources freely available11 for full reproducibility (Goodman et al., 2016).”
BackgroundPredator Control Needs a Standard of Unbiased Randomized Experiments With Cross-Over Design
Treves, A., Krofel, M., Ohrens, O., & van Eeden, L. M. (2019) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2019.00462 · Cited by 70
Mountford, N., & Cai, Y. (2022) · International Journal of Management Reviews · doi:10.1111/ijmr.12313 · Cited by 66
“Our goal was to avoid compounding any potential subjectivity within the reviewed papers with our own inadvertent bias in paper selection and, in doing so, offer increased methodological transparency (Aguinis et al., 2018) and increased inferential reproducibility (Goodman et al., 2016).”
BackgroundPromoting trust in research and researchers: How open science and research integrity are intertwined
Haven, T. L., Gopalakrishna, G., Tijdink, J. K., van der Schot, D., & Bouter, L. M. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06169-y · Cited by 90
Renton, A., Dao, T. T., Johnstone, T., Civier, O., Sullivan, R. P., White, D., Lyons, P., Slade, B., Abbott, D. F., Amos, T. J., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02145-x · Cited by 83
Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing
Penders, B., Holbrook, J. B., & de Rijcke, S. (2019) · Publications · doi:10.3390/publications7030052 · Cited by 80
“Although terms such as ‘replication’, ‘reproduction’, ‘robustness’, and ‘generalisability’ are common terminology in discussions of scientific quality control, they are not defined consistently and sometimes not defined at all [10,11].”
BackgroundNAFLD Preclinical Models: More than a Handful, Less of a Concern?
Oligschlaeger, Y., & Shiri‐Sverdlov, R. (2020) · Biomedicines · doi:10.3390/biomedicines8020028 · Cited by 74
Unreproducible Research is Reproducible
Bouthillier, X., Laurent, C., & Vincent, P. (2019) · International Conference on Machine Learning · Cited by 66
Reproducible Research: A Retrospective
Peng, R. D., & Hicks, S. C. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-012420-105110 · Cited by 89
“The terminology of reproducible research can be bewildering to some in the scientific community because there is little agreement about the meaning of the phrase in relation to other related concepts (3, 21, 35).”
Allkja, J., van Charante, F., Aizawa, J., Reigada, I., Guarch-Pérez, C., Vazquez-Rodriguez, J. A., Cos, P., Coenye, T., Fallarero, A., Zaat, S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-93115-w · Cited by 63
Annual Research Review: Translational machine learning for child and adolescent psychiatry
Dwyer, D., & Koutsouleris, N. (2022) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13545 · Cited by 76
Meta-Analysis of Complex Interventions
Tanner-Smith, E. E., & Grant, S. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-014112 · Cited by 62
Semantic Earth Observation Data Cubes
Augustin, H., Sudmanns, M., Dirk Tiede, A., Lang, S., & Baraldi, A. (2019) · Data · doi:10.3390/data4030102 · Cited by 66
“While these platforms are powerful, analyses sometimes have limited transferability to di ff erent geographic locations or points in time, or a low level of results or inferential reproducibility [22].”
BackgroundA browser-based tool for visualization and analysis of diffusion MRI data
Yeatman, J. D., Richie-Halford, A., Smith, J. K., Keshavan, A., & Rokem, A. (2018) · Nature Communications · doi:10.1038/s41467-018-03297-7 · Cited by 67
CONSORT 2010 statement: extension checklist for reporting within person randomised trials
Pandis, N., Chung, B., Scherer, R. W., Elbourne, D. R., & Altman, D. G. (2019) · British Journal of Dermatology · doi:10.1111/bjd.17239 · Cited by 66
Reproducible image handling and analysis
Miura, K., & Nørrelykke, S. F. (2021) · The EMBO Journal · doi:10.15252/embj.2020105889 · Cited by 65
“To fix our nomenclature, we will follow Goodman et al ( 2016 ), who distinguished three types of reproducibility in terms of experiments and data analyses: 1 Methods reproducibility: The original meaning of reproducibility (Claerbout & Karrenbach, 1992 ).”
MethodThe frequency of cortical microstimulation shapes artificial touch
Callier, T., Brantly, N. W., Caravelli, A., & Bensmaïa, S. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1916453117 · Cited by 66
Increasing the Impact of Materials in and beyond Bio-Nano Science
Björnmalm, M., Faria, M., & Caruso, F. (2016) · Journal of the American Chemical Society · doi:10.1021/jacs.6b08673 · Cited by 67
A critical reexamination of doing arithmetic nonconsciously
Moors, P., & Hesselmann, G. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1292-x · Cited by 66
“Within the conceptual framework proposed by Goodman et al. (2016), our reanalyses therefore suggest low inferential reproducibility of the study by Sklar et al. A critical reviewer suggested that, based on the results of our reanalyses, one would expect that the original findings would not replicate easily. In this co…”
ResultImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2018) · Brain and Behavior · doi:10.1002/brb3.1141 · Cited by 80
Artner, R., Verliefde, T., Steegen, S., Gomes, S. L. B. T., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2020) · Psychological Methods · doi:10.1037/met0000365 · Cited by 79
“An in-depth discussion of terminology in different scientific fields can be found in Goodman et al. (2016).”
BackgroundProtect us from poor-quality medical research
ESHRE Capri Workshop Group, Ioannidis, J. P. A., Bhattacharya, S., Evers, J. L. H., der Veen, F. V., Somigliana, E., Barratt, C. L. R., Bontempi, G., Baird, D. T., Crosignani, P., et al. (2018) · Human Reproduction · doi:10.1093/humrep/dey056 · Cited by 59
Self-citationReporting guidelines used varying methodology to develop recommendations
Schlüssel, M. M., Sharp, M. K., de Beyer, J. A., Kirtley, S., Logullo, P., Dhiman, P., MacCarthy, A., Koroleva, A., Speich, B., Bullock, G. S., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.03.018 · Cited by 62
Biomedical researchers’ perspectives on the reproducibility of research
Cobey, K. D., Ebrahimzadeh, S., Page, M. J., Thibault, R. T., Nguyen, P., Abu-Dalfa, F., & Moher, D. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002870 · Cited by 80
“Terminology around reproducibility varies [5]; here, we define reproducibility as re-doing a study using similar methods and obtaining findings consistent with the original study and as irreproducible when the findings are not consistent with the original study.”
ResultHoffmann, S., Schönbrodt, F. D., Elsas, R., Wilson, R., Strasser, U., & Boulesteix, A. (2021) · Royal Society Open Science · doi:10.1098/rsos.201925 · Cited by 83
“Instead, the problems seem to be more subtle and partly due to the multiplicity of possible analysis strategies [16, 17].”
BackgroundLekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58
“The endeavour of documentation also includes the decisions made by the AI system [56], to enable identifying the reasons why an AI-decision was erroneous, which, in turn, could help prevent future mistakes.”
BackgroundBrown, A. W., Aslibekyan, S. W., Bier, D. M., da Silva, R. F., Hoover, A. W., Klurfeld, D. M., Loken, E., Mayo‐Wilson, E., Menachemi, N., Pavela, G., et al. (2021) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2021.1985427 · Cited by 65
“…of well-characterized discordant twins and diet and mortality in larger samples linked to national health registers (Naukkarinen et al. 2012; Granic et al. 2013), and exploring the association of exposure to breastfeeding with obesity in childhood and adolescence using multiple sibling…”
BackgroundStructural and functional connectivity reconstruction with CATO - A Connectivity Analysis TOolbox
de Lange, S. C., Helwegen, K., & van den Heuvel, M. P. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120108 · Cited by 56
A systematic review on machine learning in sellar region diseases: quality and reporting items
Qiao, N. (2019) · Endocrine Connections · doi:10.1530/ec-19-0156 · Cited by 59
LitSense: making sense of biomedical literature at sentence level
Allot, A., Chen, Q., Kim, S., Alvarez, R. V., Comeau, D. C., Wilbur, W. J., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz289 · Cited by 56
“Having similar findings from independent studies is common in research and important to assess reproducibility (33).”
ResultReproducibility and Replicability in a Fast-Paced Methodological World
Epskamp, S. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847421 · Cited by 67
(Re)considering the Concept of Literature Review Reproducibility
Cram, W. A., Templier, M., Laval, U., Paré, G., & Montréal, H. (2020) · Journal of the Association for Information Systems · doi:10.17705/1jais.00630 · Cited by 80
“Secondly, it distinguishes this activity from the concept of reproducibility, in terms of findings that could be duplicated by an independent party, based on existing data (e.g., Bollen et al., 2015; Cassey & Blackburn, 2006; Goodman et al., 2016; Peng, 2011).”
BackgroundMethodInfluentialBiosecurity in an age of open science
Smith, J. A., & Sandbrink, J. B. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001600 · Cited by 64
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 83
Güttinger, S. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-019-0269-1 · Cited by 84
Generalizable Biomarkers in Critical Care: Toward Precision Medicine
Sweeney, T. E., & Khatri, P. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002402 · Cited by 55
Hutton, B., Wolfe, D. M., Moher, D., & Shamseer, L. (2017) · Evidence-Based Mental Health · doi:10.1136/eb-2017-102666 · Cited by 57
“In a recent editorial, Goodman et al address the issue of inconsistent views on what the term ‘reproducibility ’ means.(4) In short, it refers to the transparency and reliability of research, and whether published methods can be repeated to yield the same results and conclusions as originally reported.”
ResultDougherty, M. R. P., Slevc, L. R., & Grand, J. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691618810693 · Cited by 77
Replicability and replication in the humanities
Peels, R. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-018-0060-4 · Cited by 80
Fusaroli, R., Grossman, R. B., Bilenberg, N., Cantio, C., Jepsen, J. R. M., & Weed, E. (2021) · Autism Research · doi:10.1002/aur.2661 · Cited by 68
“…of that finding is the production of similar results (same direction of effects and comparable effect sizes) in a new study following analogous experimental and/or statistical procedures (Goodman et al., 2016), while not finding the same pattern of results indicates a failed replication.”
ResultSteele, B., Martin, M., Yakubovich, A. R., Humphreys, D. K., & Nye, E. (2020) · Trauma Violence & Abuse · doi:10.1177/1524838020970900 · Cited by 70
“…number of factors in one study (even if they were prespecified) as this can lead to “p-hacking,” cherry-picking, multiplicity (multiple variables to measure the same factor), and generating hypotheses after the results, in attempts to find statistically significant findings (Goodman et al., 2016).”
ResultOpen science, the replication crisis, and environmental public health
Hicks, D. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1962713 · Cited by 63
“S. N. Goodman, Fanelli, and Ioannidis (2016) initially follow the definitions from Bollen et al. (2015), then complain that these “definitions do not provide clear operational criteria for what constitutes successful replication or reproduction”; they go on to distinguish three senses of…”
BackgroundMethodMice in translational neuroscience: What R we doing?
Azkona, G., & Sánchez‐Pernaute, R. (2022) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2022.102330 · Cited by 62
Smith, M. M., Clarke, E., & Little, C. B. (2016) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2016.10.016 · Cited by 55
Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses
Pradhan, R., Hoaglin, D. C., Cornell, M., Liu, W., Wang, V., & Yu, H. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.08.023 · Cited by 55
“(26) Finally, EXACT can serve as a model for developing applications to extract data from other clinical trial registries that harbor results databases (like the European Union Clinical Trials Register).”
MethodHardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M. C., Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., et al. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/39cfb · Cited by 64
Synaptic Loss in Frontotemporal Dementia Revealed by [ 11 C ] UCB‐J Positron Emission Tomography
Malpetti, M., Jones, P. S., Cope, T. E., Holland, N., Naessens, M., Rouse, M. A., Rittman, T., Savulich, G., Whiteside, D. J., Street, D., et al. (2023) · Annals of Neurology · doi:10.1002/ana.26543 · Cited by 53
Hukku, A., Sampson, M. G., Luca, F., Piqué-Regi, R., & Wen, X. (2022) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2022.04.005 · Cited by 53
Computational reproducibility of Jupyter notebooks from biomedical publications
Samuel, S., & Mietchen, D. (2024) · GigaScience · doi:10.1093/gigascience/giad113 · Cited by 91
“A potential solution to this confusion is the proposed distinction [27] between Methodsreproducibility (providing enough detail about the originalstudythattheproceduresanddatacanberepeatedexactly), Resultsreproducibility (obtaining the same results when matching the original procedures and data as…”
Computational and Informatic Advances for Reproducible Data Analysis in Neuroimaging
Poldrack, R. A., Gorgolewski, K. J., & Varoquaux, G. P. (2019) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-072018-021237 · Cited by 61
Principles for data analysis workflows
Stoudt, S., Vásquez, V., & Martinez, C. C. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008770 · Cited by 64
“Despite various calls for the creation of a standard framework [7,46], achieving complete reproducibility may go far beyond the individual researcher to encompass a culture-wide shift in expectations by consumers of scientific research products, to realistic capacities of version control software.”
BackgroundMethodvan Zelst, I., Crameri, F., Pusok, A. E., Glerum, A., Dannberg, J., & Thieulot, C. (2022) · Solid Earth · doi:10.5194/se-13-583-2022 · Cited by 66
“different researchers can obtain similar enough results from independent development) (Bollen et al., 2015; Goodman et al., 2016).”
ResultLogullo, P., de Beyer, J. A., Kirtley, S., Schlüssel, M. M., & Collins, P. G. S. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-112126 · Cited by 64
Validity Concerns in Research Using Organic Data
Xu, H., Zhang, N., & Zhou, L. (2019) · Journal of Management · doi:10.1177/0149206319862027 · Cited by 59
“…psychology) under different terms, like reproducibility, replicability, conceptual cross-validation, methodological triangulation , and so on, which were often used interchangeably without a consensus on their definitions across different research fields (see Goodman, Fanelli, & Ioannidis, 2016).”
BackgroundSeyfried, T. N., & Chinopoulos, C. (2021) · Metabolites · doi:10.3390/metabo11090572 · Cited by 59
Trends in big data analyses by multicenter collaborative translational research in psychiatry
Onitsuka, T., Hirano, Y., Nemoto, K., Hashimoto, N., Kushima, I., Koshiyama, D., Koeda, M., Takahashi, T., Noda, Y., Matsumoto, J., et al. (2021) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13311 · Cited by 55
“First, to secure the compatibility of research independence and complementarity among institutions in clinical research, it is critical to examine the reproducibility of data obtained from samples at one institution using data from other institutions.(3,4) For example, in COCORO, in addition to using data from other i…”
MethodFaggion, C. M., Huivin, R., Aranda, L., Pandis, N., & Alarcon, M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.02.011 · Cited by 49
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.