The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 401–500 of 1,003
The Impact of Nondeterminism on Reproducibility in Deep Reinforcement Learning
Nagarajan, P., Warnell, G., & Stone, P. (2018) · Cited by 20
Factors that impact fragility index and their visualizations
Lin, L. (2020) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13428 · Cited by 22
The assessment of intrinsic credibility and a new argument for p < 0.005
Held, L. (2019) · Royal Society Open Science · doi:10.1098/rsos.181534 · Cited by 20
Standing on the Feet of Giants — Reproducibility in AI
Gundersen, O. E. (2019) · AI Magazine · doi:10.1609/aimag.v40i4.5185 · Cited by 19
“Replication, as used by (Drummond 2009) and (Stodden 2011), is in line with methods reproducibility as proposed by (Goodman et al. 2016) while reproducibility seems to entail both results reproducibility and inferential reproducibility.”
MethodAssessing the Robustness of Mediation Analysis Results Using Multiverse Analysis
Rijnhart, J. J. M., Twisk, J. W. R., Deeg, D. J. H., & Heymans, M. W. (2021) · Prevention Science · doi:10.1007/s11121-021-01280-1 · Cited by 27
Statistical Analysis of Data Repeatability Measures
Wang, Z., Bridgeford, E., Wang, S., Vogelstein, J. T., & Caffo, B. S. (2024) · International Statistical Review · doi:10.1111/insr.12591 · Cited by 29
“General reviews of the concept of research reproducibility, with comparison to replicability, can be found in Goodman et al. (2016) & Patil et al. (2016).”
Reasons for qualitative psychologists to share human data
Karhulahti, V. (2022) · British Journal of Social Psychology · doi:10.1111/bjso.12573 · Cited by 24
Towards a more realistic interpretation of the human fossil record
Wood, B. A., & Smith, R. J. (2022) · Quaternary Science Reviews · doi:10.1016/j.quascirev.2022.107722 · Cited by 25
Luijken, K., Lohmann, A., Alter, U., Gonzalez, J. C., Clouth, F. J., Fossum, J. L., Hesen, L., Huizing, A., Ketelaar, J., Montoya, A. K., et al. (2024) · Royal Society Open Science · doi:10.1098/rsos.231003 · Cited by 24
“, we assume that results turn out similar if simulated data, implemented computations, and software functionalities are similar (Clemens, 2017; Goodman et al., 2016).”
BackgroundReproducibility in Biomarker Research and Clinical Development: A Global Challenge
Scherer, A. (2017) · Biomarkers in Medicine · doi:10.2217/bmm-2017-0024 · Cited by 19
In-depth examination of spatiotemporal figures in open reproducible research
Konkol, M., & Kray, C. (2018) · Cartography and Geographic Information Science · doi:10.1080/15230406.2018.1512421 · Cited by 23
Trensch, G., Gutzen, R., Blundell, I., Denker, M., & Morrison, A. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2018.00081 · Cited by 19
“To be more specific about the terminology of reproducibility, in this study we aim for results reproducibility (Goodman et al., 2016; see also Plesser, 2018).”
BackgroundBouffartigue, C., Debille, S., Fabreguettes, O., Ramos‐Cabrer, A. M., Pereira‐Lorenzo, S., Flutre, T., & Harvengt, L. (2020) · Annals of Forest Science · doi:10.1007/s13595-020-00982-w · Cited by 19
“To facilitate method reproducibility (Goodman et al. 2016), most of our analyses were performed in R (R Core Team 2019); the scripts are available on the server data.inrae.fr (Bouffartigue et al., 2020).”
MethodReplication and the Establishment of Scientific Truth
Iso‐Ahola, S. E. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.02183 · Cited by 29
Transparency About Values and Assertions of Fact in Natural Resource Management
Treves, A., Paquet, P. C., Artelle, K. A., Cornman, A. M., Krofel, M., & Darimont, C. T. (2021) · Frontiers in Conservation Science · doi:10.3389/fcosc.2021.631998 · Cited by 19
“…own reproducibility crisis like that of many other scientific disciplines (Open Science Collaboration, 2015; Allison et al., 2016; Baker and Brandon, 2016; Goodman et al., 2016; Munafò et al., 2017; Clark and Alvino, 2018), but with potential consequences for the persistence of wild populations.”
BackgroundMayo‐Wilson, E., Grant, S., & Supplee, L. H. (2021) · Prevention Science · doi:10.1007/s11121-021-01284-x · Cited by 26
“Multiplehypothesis testing and selective non-reporting of results threaten the truthfulness of published research (Goodman et al., 2016).”
BackgroundTimothy, M., & Forlano, P. M. (2019) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2019.04.009 · Cited by 19
“In line with the core theme of our plugin, an emphasis on record keeping facilitates reviewing for consistency in analysis procedures and ultimately ensures methodological reproducibility (Goodman et al., 2016).”
Essers, P., van der Heijden, M., Verhagen, C. V. M., Ploeg, E. M., de Roest, R. H., Leemans, C. R., Brakenhoff, R. H., van den Brekel, M. W. M., Bartelink, H. G. M. M., Verheij, M., et al. (2019) · Cancer Research · doi:10.1158/0008-5472.can-18-3388 · Cited by 21
Steegmans, P. A. J., Girolamo, N. D., & Reynders, R. M. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0084-4 · Cited by 19
Knock, E., Knock, E., Perez‐Guzman, P. N., Bhatia, S. N., Guntoro, F., Guntoro, F., Whittaker, C., Ferguson, N. M., Cori, A., Baguelin, M., et al. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16466.2 · Cited by 21
Lin, C., Yang, Y. E., Malek, K., & Adam, J. C. (2022) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2022.105451 · Cited by 20
“…Py-RAMID package for embracing the Open Science by Design concept (NASEM, 2018; U.S. DOE, 2019) through improving coupled models ’ reproducibility (Goodman et al., 2016) and extensibility (Lacroix and Critchlow, 2003), we understand Py-RAMID has its limitation to fully meet the idea due to the…”
MethodImbrogno, A., Nguyen, M. N., & Schäfer, A. I. (2024) · Chemosphere · doi:10.1016/j.chemosphere.2024.141833 · Cited by 18
Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
Lombaers, P., de Bruin, J., & van de Schoot, R. (2024) · Applied Sciences · doi:10.3390/app14093842 · Cited by 20
“In Goodman, Fanelli [36], the authors try to clarify the reproducibility terminology by distinguishing between three different kinds of reproducibility depending on the goal: ‘method reproducibility’, ‘results reproducibility’, and ‘inferential reproducibility’.”
InfluentialBotallo's error, or the quandaries of the universality assumption
Bartolomeo, P., Malkinson, T. S., & Vito, S. D. (2016) · Cortex · doi:10.1016/j.cortex.2016.09.026 · Cited by 20
Vachon, B., Curran, J., Karunananthan, S., Brehaut, J., Graham, I. D., Moher, D., Sales, A. E., Straus, S. E., Fiander, M., Paprica, P. A., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.006 · Cited by 25
Jaeger, S. R., Roigard, C. M., Ryan, G. S., Jin, D., & Giacalone, D. (2020) · Food Quality and Preference · doi:10.1016/j.foodqual.2020.104057 · Cited by 20
Alba, S., Lenglet, A., Verdonck, K., Roth, J., Patil, R., Mendoza, W., Juvekar, S. K., & Rumisha, S. F. (2020) · BMJ Global Health · doi:10.1136/bmjgh-2020-003237 · Cited by 22
A quality framework for statistical algorithms
Yung, W., Tam, S., Buelens, B., Chipman, H., Dumpert, F., Ascari, G., Rocci, F., Burger, J., & Choi, I. (2021) · Statistical Journal of the IAOS · doi:10.3233/sji-210875 · Cited by 24
“It is important to recognize the three dimensions of reproducibility, namely, methods reproducibility, results reproducibility and inferential reproducibility [30].”
Quantitative Assessment of Grapevine Wood Colonization by the Dieback Fungus Eutypa lata
Moisy, C., Berger, G., Flutre, T., Cunff, L. L., & Péros, J. (2017) · Journal of Fungi · doi:10.3390/jof3020021 · Cited by 18
Towards computational reproducibility: researcher perspectives on the use and sharing of softwarePreprint
AlNoamany, Y., & Borghi, J. (2018) · doi:10.7287/peerj.preprints.26727v1 · Cited by 18
Replication and methodological robustness in quantitative typology
Becker, L., & Naranjo, M. G. (2025) · Linguistic Typology · doi:10.1515/lingty-2023-0076 · Cited by 18
“We will de fi ne robustness following Goodman et al. (2016) as shown in (1).”
Payne, P., Lele, O., Johnson, B. H., & Holve, E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6937 · Cited by 18
“The principles and practices of Open Science offer a promising path to address both goals by facilitating (1) increased transparency of data and methods, which promotes research reproducibility and rigor [1-4]; and (2) cumulative efficiencies wherein research tools and the output of research are combined to accelerate…”
Usée, F., Jacobs, A. M., & Lüdtke, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00905 · Cited by 27
Current use of effect size or confidence interval analyses in clinical and biomedical research
Amaral, E. D. O. S., & Line, S. R. P. (2021) · Scientometrics · doi:10.1007/s11192-021-04150-3 · Cited by 23
Expanding the Immunology Toolbox: Embracing Public-Data Reuse and Crowdsourcing
Sparks, R., Lau, W. W., & Tsang, J. S. (2016) · Immunity · doi:10.1016/j.immuni.2016.12.008 · Cited by 19
Statistical methods for replicability assessment
Hung, K. E., & Fithian, W. (2020) · The Annals of Applied Statistics · doi:10.1214/20-aoas1336 · Cited by 26
Improving Process Descriptions in Research by Model-Based Analysis
Shaked, A., & Reich, Y. (2021) · IEEE Systems Journal · doi:10.1109/jsyst.2020.2990488 · Cited by 17
“[47]—relates to inferential reproducibility”
Scientific workflow managers in metabolomics: an overview
Verhoeven, A., Giera, M. A., & Mayboroda, O. A. (2020) · The Analyst · doi:10.1039/d0an00272k · Cited by 17
Quantity Over Quality? Reproducible Psychological Science from a Mixed Methods Perspective
Reischer, H. N., & Cowan, H. R. (2020) · Collabra Psychology · doi:10.1525/collabra.284 · Cited by 24
Life Stressors: Elevations and Disparities Among Older Adults with Pain
Brennan, P. L. (2020) · Pain Medicine · doi:10.1093/pm/pnaa189 · Cited by 18
“As yet there is no consensus on the best methods to judge successful reproduction of study results [72], but according to one criterion established by the Open Science Collaboration [73], shared statistical significance of effects, this investigation clearly demonstrated substantial overlap between findings obtained i…”
MethodIs Critical Human Geography Research Replicable?
Wainwright, J. D. (2020) · Annals of the American Association of Geographers · doi:10.1080/24694452.2020.1806025 · Cited by 19
“Like most social science (Goodman, Fanelli, and Ioannidis 2016; Servick 2018; National Academy of Sciences 2019), very little contemporary research in human geography is reproducible or, indeed, reproduced.”
BackgroundBiofilms vs. cities and humans vs. aliens - a tale of reproducibility in biofilms.
Azevedo, N., Allkja, J., & Goeres, D. (2021) · Trends in Microbiology · doi:10.1016/j.tim.2021.05.003 · Cited by 18
“There is also a scientific cost as subsequent research is often based on conclusions from preclinical studies, and the lack of reproducibility could result in dead-end studies [6,7].”
BackgroundWriting a review article - Are you making these mistakes?
Daldrup-Link, H. E. (2018) · Nanotheranostics · doi:10.7150/ntno.24793 · Cited by 17
Zaragozí, B., Trilles, S., & Carrión, J. T. N. (2020) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9030138 · Cited by 20
“) are based on open and publicly accessible resources that can produce the same results as those that have been published [18].”
BackgroundRe-evaluating Word Mover's DistancePreprint
Sato, R., Yamada, M., & Kashima, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14403 · Cited by 20
“The importance of reproducing existing experiments by third-party groups has been widely recognized in science (Lin, 2018; Sculley et al., 2018; Munafò et al., 2017; Collins & Tabak, 2014; Goodman et al., 2016).”
BackgroundDocumentation strategy for facilitating the reproducibility of geo-simulation experiments
Zhu, Z., Chen, M., Qian, Z., Li, H., Wu, K., Ma, Z., Wen, Y., Yue, S., & Lü, G. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105687 · Cited by 16
Reito, A. R., Raittio, L., & Helminen, O. (2019) · PeerJ · doi:10.7717/peerj.6813 · Cited by 16
Seven Reasons Why: A User’s Guide to Transparency and Reproducibility
Filho, D. B. F., Lins, R. G. P., Domingos, A. R. M., Janz, N., & Silva, L. (2019) · Brazilian Political Science Review · doi:10.1590/1981-3821201900020001 · Cited by 24
“Goodman, Fanelli and Ioannidis (2016) argue that the current use of reproducible research was initially employed in computational science not for corroboration but transparency.”
BackgroundOn dose cube pixel spacing pre-processing for features extraction stability in dosiomic studies
Placidi, L., Cusumano, D., Lenkowicz, J., Boldrini, L., & Valentini, V. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.09.010 · Cited by 16
Physical activity and sleep during the first week of anorexia nervosa inpatient care
Langlet, B. S., Vestermark, F., Stolt, J., Zandian, M., Södersten, P., & Bergh, C. E. K. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260077 · Cited by 16
“The secondary objective was to increase result reproducibility, defined as “the production of corroborating results in a new study, having followed the same experimental methods” [35], and generalisability of measurements in the current population by using open-source algorithms and an accelerometer which allows extra…”
BackgroundA Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software
Salsabil, L., Wu, J. F., Choudhury, M. H., Ingram, W. A., Fox, E. A., Rajtmajer, S. M., & Giles, C. L. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524658 · Cited by 20
“rials as were used by the original investigators [1, 7].”
MethodA blockchain based Trusted Persistent Identifier system for Big Data in Science
Bellini, E. (2019) · Foundations of Computing and Decision Sciences · doi:10.2478/fcds-2019-0018 · Cited by 19
“Today, Open Science and reproducible research have become a crucial goal across research communities and funding bodies [18], [6], [32].”
BackgroundBiagi, D., Fantozzi, E. T., Campos-Oliveira, J. C., Naghetini, M. V., Ribeiro, A. F., Rodrigues, S. D. S., Ogusuku, I. E. Y., Vanderlinde, R., Christie, M. L. A., Mello, D. B., et al. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11050374 · Cited by 16
Gkrouzoudi, A., Tsingotjidou, A., & Jirkof, P. (2021) · Physiology & Behavior · doi:10.1016/j.physbeh.2021.113645 · Cited by 16
“fields, including animal based research (5), (6) and may also affect ECG studies.”
Dissimulate or disseminate? A survey on the fate of negative results
Herbet, M., Léonard, J., Santangelo, M., & Albaret, L. (2022) · Learned Publishing · doi:10.1002/leap.1438 · Cited by 17
Comparing Bayesian and Frequentist Models of Language Variation: The Case of Help + (to-)Infinitive
Levshina, N. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108589314.009 · Cited by 18
Reproducibility in Subsurface Geoscience
Steventon, M., Jackson, C., Hall, M., Ireland, M., Munafò, M., & Roberts, K. (2021) · Earth Science, Systems and Society · doi:10.3389/esss.2022.10051 · Cited by 9
GeoAI Reproducibility and Replicability: A Computational and Spatial Perspective
Li, W., Hsu, C.-Y., Wang, S., & Kedron, P. (2024) · Annals of the American Association of Geographers · doi:10.1080/24694452.2024.2373787 · Cited by 33
“Reproducibility can be further categorized into methods reproducibility and results reproducibility by distinguishing the focus of the the reproduction(Goodman, Fanelli, & Ioannidis, 2016).”
Credit and Priority in Scientific Discovery: A Scientist’s Perspective
Flier, J. (2019) · Perspectives in biology and medicine · doi:10.1353/pbm.2019.0010 · Cited by 15
Tsvetkov, D., Kolpakov, E., Kaßmann, M., Schubert, R., & Gollasch, M. (2019) · Frontiers in Medicine · doi:10.3389/fmed.2019.00126 · Cited by 17
“As such, reproducibility in biomedical research has become an important topic of debate in the scientific community (12, 13).”
BackgroundThe Academic–Industrial Complexity: Failure to Launch
Levin, L. A., & Béhar‐Cohen, F. (2017) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2017.10.003 · Cited by 15
Altenburger, K. M., & Ho, D. E. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313683 · Cited by 23
“Our evidence hence underscores recent calls for more general reproducibility (or “conceptual replication”) in machine learning [15, 20, 38].”
BackgroundInconsistency of findings due to low power: A structural MRI study of bilingualism
Munson, B. A., & Hernández, A. E. (2019) · Brain and Language · doi:10.1016/j.bandl.2019.104642 · Cited by 15
Gosselin, R. (2019) · BioEssays · doi:10.1002/bies.201900189 · Cited by 13
The analysis of living systems can generate both knowledge and illusions
Jose, A. M. (2020) · eLife · doi:10.7554/elife.56354 · Cited by 14
Heinrich, A., & Knight, S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01590 · Cited by 15
“We acknowledge that a lively discussion exists within the wider field of behavioral science regarding the theoretical question of what exactly constitutes “reproducibility” and how best to assess it (Schmidt, 2009; Goodman et al., 2016).”
BackgroundKoike, R., Cueno, M. E., Nodomi, K., Tamura, M., Kamio, N., Tanaka, H., Kotani, A., & Imai, K. (2020) · Molecules · doi:10.3390/molecules25173839 · Cited by 15
Reproducibility and replicability of rodent phenotyping in preclinical studiesPreprint
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R. T., Golani, I., Gómez-Marín, À., Heller, R., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/079350 · Cited by 16
Marwick, B., Rokem, A., & Станева, В. (2017) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/gne3w · Cited by 18
Zimmermann, J., & Bach, D. R. (2020) · Learning & Memory · doi:10.1101/lm.050211.119 · Cited by 17
Reporting In Vitro Experiments Responsibly – the RIVER RecommendationsPreprint
The RIVER working group (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/x6aut · Cited by 25
Standards, dissemination, and best practices in systems biology
Shin, J., Porubsky, V., Carothers, J., & Sauro, H. M. (2023) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2023.102922 · Cited by 15
Baker, K. C., Steiner, A. Z., Hansen, K. R., Barnhart, K. T., Cedars, M. I., Legro, R. S., Diamond, M. P., Krawetz, S. A., Usadi, R., Baker, V. L., et al. (2022) · F&S Reports · doi:10.1016/j.xfre.2022.03.003 · Cited by 14
Afifi, I., Elgendy, M. A., Abdelfatah, M., & El–Sappagh, S. (2025) · Brain Informatics · doi:10.1186/s40708-025-00286-7 · Cited by 17
Reproducibility, preservation, and access to research with ReproZip and ReproServer
Steeves, V., Rampin, R., & Chirigati, F. (2020) · IASSIST Quarterly · doi:10.29173/iq969 · Cited by 13
Causal effects of psychostimulants on neural connectivity: a mechanistic, randomized clinical trial
Wang, Y., Kessel, E. M., Lee, S., Hong, S., Raffanello, E., Hulvershorn, L. A., Margolis, A. E., Peterson, B. S., & Posner, J. E. (2022) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13585 · Cited by 15
Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines
Rahal, R.-M., Hamann, H., Brohmer, H., & Pethig, F. (2022) · Research Ideas and Outcomes · doi:10.3897/rio.8.e89980 · Cited by 19
Chuang, Z., Martin, J. E., Shapiro, J., Nguyen, D. L., Neocleous, P., & Jones, P. M. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2022.11.001 · Cited by 14
Kelter, R. (2023) · Biometrical Journal · doi:10.1002/bimj.202200095 · Cited by 21
New Insights into User-Defined Smart Ring Gestures with Implications for Gesture Elicitation Studies
Gheran, B.-F., Vatavu, R.-D., & Vanderdonckt, J. M. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585590 · Cited by 14
“The RepliGES space was built on top of the replication literature [15,16,29,36], from which it adopted methods and provided terminology adaptations to speak directly to HCI practitioners and help identify replication types relevant for gesture elicitation studies.”
MethodUsing negative controls to adjust for unmeasured confounding bias in time series studies
Hu, J. K., Tchetgen, E. J. T., & Dominici, F. (2023) · Nature Reviews Methods Primers · doi:10.1038/s43586-023-00249-4 · Cited by 19
Changing times for science and the public
Liskauskas, S., Ribeiro, M. D., & Vasconcelos, S. M. R. (2019) · EMBO Reports · doi:10.15252/embr.201947906 · Cited by 19
“One issue is that even the notion of “reproducible research” should not be taken for granted, as standards, methodologies, and expectations vary widely across different research fields [6,7].”
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 16
Bachhofner, S., Loghin, A.-M., Otepka, J., Pfeifer, N., Hornáček, M., Siposova, A., Schmidinger, N., Hornik, K., Schiller, N., Kähler, O., et al. (2020) · Remote Sensing · doi:10.3390/rs12081289 · Cited by 15
“This controversy invoked a discussion about the question “What does research reproducibility mean?” [139] and a discussion about how scientists can avoid fooling themselves [140].”
BackgroundLubega, N., Anderson, A., & Nelson, N. C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0293584 · Cited by 16
“This growing literature has focused on identifying the specific types of problems that may give rise to irreproducibility (such as publication bias or lack of masking) [4,5], quantifying how often these problems arise [6–8], and proposing and evaluating potential solutions [9–11].”
BackgroundTriple low, double low: it’s time to deal Achilles heel a single deadly blow
Willingham, M. D., & Avidan, M. S. (2017) · British Journal of Anaesthesia · doi:10.1093/bja/aex132 · Cited by 14
Repeatability and reproducibility of longitudinal relaxation rate in 12 small-animal MRI systems
Waterton, J. C., Hines, C. D. G., Hockings, P. D., Laitinen, I., Ziemian, S., Campbell, S. P., Gottschalk, M. E., Green, C., Haase, M. V., Hassemer, K., et al. (2019) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2019.03.008 · Cited by 14
“More generally, it is important to demonstrate reproducibility for multiple animal studies in different laboratories [26] to address the perceived “reproducibility crisis” [27] in translational medicine [28].”
BackgroundNørgaard, M., Ganz, M., Svarer, C., Frøkjær, V. G., Greve, D. N., Strother, S. C., & Knudsen, G. M. (2019) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x19880450 · Cited by 17
“available equipment, settings, and sample data.(3) Apart from differences in methodology, it is also challenging to identify the sources of variation that originate from each methodological choice, and how it may ultimately influence the study outcome.”
BackgroundImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2018) · Cancer Reports · doi:10.1002/cnr2.1150 · Cited by 21
“Without a rigorous description of the methods, materials, and statistical approaches, experimenters lack the necessary information to independently replicate or nearly replicate results with the same protocol under similar conditions (Goodman et al., 2016; Kilkenny et al., 2009).”
InfluentialRepeatability and Reproducibility of SMTube Measurement in Dry Eye Disease Patients
Hao, Y., Tian, L., Cao, K., & Jie, Y. (2021) · Journal of Ophthalmology · doi:10.1155/2021/1589378 · Cited by 14
Reynders, R. M., & Ladu, L. (2017) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6401257 · Cited by 14
Heyard, R., Pawel, S., Frese, J., Voelkl, B., Würbel, H., McCann, S., Held, L., Wever, K. E., Hartmann, H., Townsin, L., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242076 · Cited by 20
Reproducibility and robustness of economics and political science research
Brodeur, A., Mikola, D., Cook, N., Fiala, L., Brailey, T., Briggs, R. C., de Gendre, A., Dupraz, Y., Gabani, J., Gauriot, R., et al. (2026) · Nature · doi:10.1038/s41586-026-10251-x · Cited by 34
Siex, P., Nowlin, W., Ottwell, R., Arthur, W., Checketts, J. X., Thompson, J., Small, T. J., Reddick, B., Wright, D., Hartwell, M. L., et al. (2020) · Osteoarthritis and Cartilage Open · doi:10.1016/j.ocarto.2020.100121 · Cited by 12
Statistical Perspectives on Reproducibility: Definitions and Challenges
Simkus, A., Coolen‐Maturi, T., Coolen, F. P. A., & Bendtsen, C. (2025) · Journal of Statistical Theory and Practice · doi:10.1007/s42519-025-00459-x · Cited by 15
“Goodman et al. [64] presented inferential reproducibility which leads to similar conclusions from “an independent replication of a study or a re-analysis of the original study.””
InfluentialRevisiting reproducibility in transportation simulation studies
Riehl, K., Kouvelas, A., & Makridis, M. (2025) · European Transport Research Review · doi:10.1186/s12544-025-00718-9 · Cited by 19
Norman, C. R., Leeflang, M., & Névéol, A. (2018) · Europe PMC (PubMed Central) · doi:10.5281/zenodo.2574709 · Cited by 14
Improving The Reproducibility of Science
Held, L., & Schwab, S. (2020) · Significance · doi:10.1111/j.1740-9713.2020.01351.x · Cited by 13
Valdez, D., Vorland, C. J., Brown, A. W., Mayo-Wilson, E., Otten, J. M., Ball, R. J., Grant, S., Levy, R., Svetina, D., & Allison, D. B. (2020) · F1000Research · doi:10.12688/f1000research.26594.1 · Cited by 16
“There are several types of reproducibility, which include the ability to evaluate and follow the same procedures as previous studies, obtain comparable results, and draw similar inferences (Goodman et al ., 2016; National Academies of Sciences, 2019).”
ResultThe Potassium Cycle and Its Relationship to Recommendation Development
Brouder, S. M., Volenec, J. J., & Murrell, T. S. (2020) · Springer eBooks · doi:10.1007/978-3-030-59197-7_1 · Cited by 17
“While numerous factors can contribute to irreproducible or non-generalizable results, solutions for improving the quality of science consistently stress complete reporting inclusive of data access (Begley and Ioannidis 2015; Button et al. 2013; Goodman et al. 2016).”
Background
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.