The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
What does research reproducibility mean?
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 601–700 of 1,003
The primary pharmacology of ceftazidime/avibactam: in vitro translational biology
Nichols, W. W., Bradford, P. A., Lahiri, S. D., & Stone, G. G. (2022) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkac171 · Cited by 8
Hansford, H. J., Cashin, A. G., Bagg, M. K., Wewege, M. A., Ferraro, M. C., Kianersi, S., Mayo-Wilson, E., Grant, S., Toomey, E. C., Skinner, I., et al. (2022) · Sports Medicine - Open · doi:10.1186/s40798-022-00496-x · Cited by 12
Giofrè, D., Boedker, I., Cumming, G., Rivella, C., & Tressoldi, P. E. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01993-3 · Cited by 13
“This observation has led scientists, funders, research users, and policymakers to investigate the reasons behind the absence of replicability (e.g., Goodman et al., 2016).”
BackgroundNudging transparent behavioural science and policy
Maynard, O., & Munafo, M. R. (2018) · Behavioural Public Policy · doi:10.1017/bpp.2018.10 · Cited by 9
Regarding the alterations in oxidative stress status induced by melasma treatments
Espósito, A. C. C., Cassiano, D. P., Bagatin, E., & Miot, H. A. (2021) · Archives of Dermatological Research · doi:10.1007/s00403-021-02205-2 · Cited by 8
Hoffmann, S., Schönbrodt, F. D., Elsas, R., Wilson, R., Strasser, U., & Boulesteix, A. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/afb9p · Cited by 8
Westlake, H. E., David, F. P. A., Tian, Y., Krakovic, K., Sah, P. K., Klotz, N., Dolgikh, A., Juravlev, L., de Bournonville, T. E., Carboni, A., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.07.663442 · Cited by 9
“Reproducibility projects are hampered by the fact that verification must be more stringent and convincing than the initial results to be widely accepted (Errington et al., 2021; Goodman et al., 2016; Shiffrin et al., 2018).”
Ngu, M. S., Toro, S., Burger, A., & Ang, K. C. (2025) · Disease Models & Mechanisms · doi:10.1242/dmm.052441 · Cited by 9
“Researchers working hands-on with data and standards in their studies provide invaluable feedback that drives iterative refinements to accommodate new research questions, contexts, or technologies (Goodman et al., 2016).”
Schueller, S., Mikelis, F., Eliades, T., & Koletsi, D. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae064 · Cited by 8
“There is an increasing interest of researchers in transparent research practices that ensure credible and reproducible research [1, 2].”
Ferreira, M., Batista, S., Camerotte, R., Vilardo, M., Nogueira, B. V., Hong, A., Santos, A. B., Günkan, A., Bocanegra-Becerra, J., Ribeiro, F., et al. (2024) · Operative Neurosurgery · doi:10.1227/ons.0000000000001440 · Cited by 8
Evaluating the method reproducibility of deep learning models in biodiversity research
Ahmed, W., Kommineni, V. K., König-Ries, B., Gaikwad, J., Gadelha, L., & Samuel, S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2618 · Cited by 9
“Method reproducibility, as mentioned in Goodman, Fanelli & Ioannidis (2016) , is an important aspect of progress in any scienti fi c fi eld.”
Engebretsen, E., Greenhalgh, T., & Han, P. K. J. (2025) · Medical Humanities · doi:10.1136/medhum-2025-013292 · Cited by 8
Is There a Reproducibility Crisis? On the Need for Evidence-based Approaches
Pethick, S., Wass, M. N., & Michaelis, M. (2025) · International Studies in the Philosophy of Science · doi:10.1080/02698595.2025.2538937 · Cited by 11
“…as obstacles to the development of concrete, constructive approaches for the meaningful investigation of different forms of data reproducibility (Goodman, Fanelli, and Ioannidis 2016; Simkus et al. 2025), and a precise, unambiguous, and applicable terminology remains needed to identify concrete…”
Repositories for academic products/outputs: Latin American and Chilean visions
Torres-Díaz, L., & Hartley, R. (2019) · F1000Research · doi:10.12688/f1000research.19976.1 · Cited by 6
Transparency and Replication in Brazilian Political Science: A First Look
Avelino, G., DESPOSATO, S. W., & Mardegan, I. O. (2021) · Dados · doi:10.1590/dados.2021.64.3.242 · Cited by 8
Reporting guidelines of Chinese medicine: Current situation and future development
Zhang, X., Chung, W. C., Lau, C. T., & Wang, N. (2022) · Journal of Traditional Chinese Medical Sciences · doi:10.1016/j.jtcms.2022.06.008 · Cited by 7
Rigor and reproducibility instruction in academic medical libraries
LaPolla, F. W. Z., Bakker, C., Exner, N., Mentnech, T., Surkis, A., & Ye, H. (2022) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2022.1443 · Cited by 6
Exogenous female sex steroid hormones and new-onset asthma in women: a matched case–control study
Zhang, G., Basna, R., Mathur, M., Lässer, C., Mincheva, R., Ekerljung, L., Wennergren, G., Rådinger, M., Lundbäck, B., Kankaanranta, H., et al. (2023) · BMC Medicine · doi:10.1186/s12916-023-03038-8 · Cited by 5
“Furthermore, Bayesian analysis allows us to make intuitive probabilistic statements about the parameters [19, 110].”
The Human Right to Science: History, Development, and Normative Content
Romano, C. P. R., & Boggio, A. (2024) · Oxford University Press eBooks · doi:10.1093/law/9780197768990.001.0001 · Cited by 10
Defining Research Reproducibility: What Do You Mean?
Mendoza, D., & Garcia, C. A. (2017) · Clinical Chemistry · doi:10.1373/clinchem.2017.279984 · Cited by 8
Schreijenberg, M., Chiarotto, A., Mauff, K. A. L., Lin, C. C., Maher, C. G., & Koes, B. W. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.010 · Cited by 7
“Although method reproducibility and inferential reproducibility have now been addressed for the PACE trial, results reproducibility (also called replication) has not [8,10,11].”
BackgroundMethodChakraborty, J., Maltzahn, C., & Jimenez, I. (2020) · International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC · doi:10.1109/canopiehpc51917.2020.00007 · Cited by 4
“Numerous existing research has tried to address the problem of reproducibility [7] by different means; for example, logging and tracing system calls, using workflow engines, using correctly provisioned shared and public testbeds, by recording and replaying changes from a stable initial state, among…”
MethodReproducibility: A Researcher-Centered Definition
Feger, S. S., & Woźniak, P. W. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6020017 · Cited by 10
“These terms are not standardized [1], are used interchangeably, carry different meanings across scientific fields, or imply questionable dependencies [2], in turn adding to the complexity of supporting and conducting responsible modern research.”
BackgroundAnalytic transparency is key for reproducibility of agricultural research
Sileshi, G. W. (2023) · CABI Agriculture and Bioscience · doi:10.1186/s43170-023-00144-8 · Cited by 8
Päll, T., Luidalepp, H., Tenson, T., & Maiväli, Ü. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002007 · Cited by 9
“…power to find true effects [7,8], in poor reproducibility (defined as getting identical results when reanalyzing the original data by the geo-original analytic workflow), and in poor replicability (defined as getting similar results after repeating the entire experiment) of the results [9].”
ResultMiller, C. T., & Rice, R. L. (2023) · Journal of Scholarly Publishing · doi:10.3138/jsp-2022-0073 · Cited by 10
Zhang, W., Liu, Y., Zhang, M., Zhu, C., & Lu, Y. (2017) · BMC Systems Biology · doi:10.1186/s12918-017-0444-y · Cited by 8
“[3] provides a survey on the papers with the word reproducibility included in titles, abstracts and keywords, and concludes that the interpretation of reproducibility varies among different papers.”
BackgroundMethodInfluentialFocused echocardiography: Dyad versus individual training in an authentic clinical context
See, K. C., Chua, J. W., Verstegen, D. M. L., van Merriënboer, J. J. G., & van Mook, W. (2018) · Journal of Critical Care · doi:10.1016/j.jcrc.2018.10.009 · Cited by 7
Turning the tables: A university league-table based on quality not quantity
Barnett, A. G., & Moher, D. (2019) · F1000Research · doi:10.12688/f1000research.18453.2 · Cited by 31
de Sousa, M. R., Cota, G., Burger, A. L., & Pezawas, T. (2021) · Journal of Cardiovascular Electrophysiology · doi:10.1111/jce.14908 · Cited by 7
Six Reasons to Consider Using R in Psychological ResearchPreprint
Barrett, T. S. (2019) · doi:10.31234/osf.io/8mb6d · Cited by 7
Dougherty, M. R. P., Slevc, L. R., & Grand, J. A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/48qux · Cited by 8
Reproducibility and Replicability in a Fast-paced Methodological WorldPreprint
Epskamp, S. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/cnq4d · Cited by 8
Renton, A., Dao, T. T., Abbott, D. F., Amos, T. J., Bollmann, S., Botting, A., Campbell, M. E. J., Chang, J., Civier, O., Close, T., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.23.521691 · Cited by 7
Coyle, J., Hejazi, N. S., Malenica, I., Phillips, R. V., Arnold, B. F., Mertens, A., Benjamin‐Chung, J., Cai, W., Dayal, S., Colford, J. M., et al. (2023) · Wiley eBooks · doi:10.1002/9781118445112.stat08414 · Cited by 8
Paolini, L., Vahdati, S., Di Iorio, A., Wardenga, R., Heibi, I., & Peroni, S. (2025) · Scientometrics · doi:10.1007/s11192-025-05418-8 · Cited by 8
Ireland, M., Algarabel, G., Steventon, M., & Munafo, M. R. (2023) · Seismica · doi:10.26443/seismica.v2i1.278 · Cited by 9
Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods
Werner, L., Layaïda, N., Genevès, P., Euzenat, J., & Graux, D. (2024) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29515 · Cited by 5
“Various terminologies have been considered for reproducibility in computer science (Drummond 2009; Goodman, Fanelli, and Ioannidis 2016; Rougier et al. 2017; Plesser 2018; Gundersen, Gil, and Aha 2018).”
MethodPetro-Petro, J., Arango-Paternina, C. M., Villada, F. A. P., Villada, J. F. R., & Brownson, R. C. (2024) · BMC Public Health · doi:10.1186/s12889-024-18615-6 · Cited by 6
“Here, replicability analysis focuses on those methodological features reported in sufficient detail so that one can ascertain whether the study may be replicated in a manner similar to the original version [4].”
Demi, S., Sánchez‐Gordón, M., Kristiansen, M., & Larrucea, X. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2707 · Cited by 6
“56 To ensure reproducibility, the research methodology is explained thoroughly in Section 3, and a sample of the analysis process can be accessed in an online repository.”
Moraes, M. A., Arabe, L. B., Resende, B. L., Codo, B. C., Reis, A. L. D. A. L., & Souza, B. R. (2025) · Hormones and Behavior · doi:10.1016/j.yhbeh.2025.105707 · Cited by 7
Chuan-Peng, H., Xu, Z., Lazić, A., Bhattacharya, P., Seda, L., Hossain, S., Jeftić, A., Özdoğru, A. A., Amaral, O. B., Miljković, N., et al. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357565 · Cited by 14
“…to duplicate the results of a prior study using the same materials/data as were used by the original investigator; “replicability” refers to the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected (Goodman et al., 2016).”
Moher, D., & Moher, E. (2016) · Annals of Internal Medicine · doi:10.7326/l16-0415 · Cited by 6
Schöch, C. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00073-y · Cited by 22
“…et al., 2010); “enhanc[ing] the reliability of research” Gomez et al. (2010); the “corroboration” of earlier evidence, claims or findings (Babin et al., 2021; Goodman et al., 2016); converting “tentative belief to accepted knowledge” (Berthon et al., 2002), also in a Bayesian perspective of…”
InfluentialOmair, A. O. M., Jabbar, A. M. A., & Albulushi, M. O. A. (2023) · International Journal of Health Sciences · doi:10.53730/ijhs.v7ns1.14680 · Cited by 8
“Through the automation of various experimental procedures, researchers may minimize variability and guarantee enhanced consistency in their findings (Goodman et al., 2016).”
BackgroundTamir, J. I., Blumenthal, M., Wang, J., Oved, T., Shimron, E., & Zaiß, M. (2025) · Magnetic Resonance Materials in Physics Biology and Medicine · doi:10.1007/s10334-025-01236-4 · Cited by 11
“Unfortunately, reproducibility, replicability, and other similar “R-words” [6] have overloaded definitions depending on the specific scientific community [7].”
Jin, X., Robinson, K., Lee, A., Polhill, G. J., Pritchard, C., & Parker, D. C. (2017) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2017.06.010 · Cited by 8
Statistical model for reproducibility in ranking-based feature selection
Urkullu, A., Pérez, A., & Calvo, B. (2020) · Knowledge and Information Systems · doi:10.1007/s10115-020-01519-3 · Cited by 7
Strupler, N. (2021) · Internet Archaeology · doi:10.11141/ia.56.6 · Cited by 7
“…same results … Reproducibility is a minimum necessary condition for a finding to be believable and informative'; • replicability - 'the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected' (see also Goodman et al. 2016).”
BackgroundSchölzel, C., Blesius, V., Ernst, G., Goesmann, A., & Dominik, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254749 · Cited by 9
“[7], with the following modeling-specific 15”
Rueda, H. G., & Bustillo, J. (2022) · Psychiatric Genetics · doi:10.1097/ypg.0000000000000309 · Cited by 7
“Results’ reproducibility refers to obtaining the same results from the conduct of an independent study that uses the same procedures (Goodman et al., 2016).”
BackgroundResultChallenges and Responses of Higher Education Institutions (HEIs) Towards Academic Resilience
Almerez, Q. L., & Duping, A. M. (2022) · International Journal of Research and Innovation in Social Science · doi:10.47772/ijriss.2022.61225 · Cited by 13
Reporting guidelines enhance the quality and impact of research in Dental Traumatology
Nagendrababu, V., Duncan, H. F., Abbott, P. V., & Dummer, P. M. H. (2023) · Dental Traumatology · doi:10.1111/edt.12824 · Cited by 7
“Moreover, it may be difficult or impossible for others to reproduce the work 8 or to compare the work quantitatively with other publications by meta-analysis.”
BackgroundResearch Integrity – Teil 1: Verantwortungsvolle Forschungspraktiken und Transparenz
Braun, T., & Kopkow, C. (2023) · physioscience · doi:10.1055/a-1982-2858 · Cited by 6
“[21] Goodman SN, Fanelli D, Ioannidis JPA.”
Ursić, L., Bralić, N., Žuljević, M. F., Puljak, L., & Buljan, I. (2024) · Accountability in Research · doi:10.1080/08989621.2024.2345723 · Cited by 8
“…and evaluation via p-values in order to advance reproducibility (Lash 2017), and is further exacerbated by the challenge of “how much evidence needs to be gathered for effective proof,” which significantly varies between disciplines and specific studies (Goodman, Fanelli, and Ioannidis 2016).”
InfluentialReproducibility and transparency in interpretive corpus pragmatics
Schweinberger, M., & Haugh, M. (2025) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.23033.sch · Cited by 8
Ferrarello, F., Viligiardi, M., & Bari, M. D. (2018) · Archives of Physiotherapy · doi:10.1186/s40945-018-0049-9 · Cited by 6
Nature, nanoscience, and textile structures
Mishra, R., & Militký, J. (2019) · Elsevier eBooks · doi:10.1016/b978-0-08-102609-0.00001-8 · Cited by 6
Blockchain Based Peer-Review Interfaces for Digital Medicine
Dhillon, V. (2020) · Frontiers in Blockchain · doi:10.3389/fbloc.2020.00008 · Cited by 9
“Rather, a complete re-thinking and re-engineering of the peer-review process with a focus on reproducibility using blockchain protocols is needed (Goodman et al., 2016).”
BackgroundBrown, R. C., Mortensen, J., Hawn, S. E., Bountress, K. E., Chowdhury, N., Kevorkian, S. S., McDonald, S. D., Pickett, T. C., Danielson, C. K., Thomas, S., et al. (2021) · Military Psychology · doi:10.1080/08995605.2021.1902183 · Cited by 6
“will be able to understand whether these differences reflect meaningful differences in underlying biopsychosocial processes or emerge as artifacts in design, measurement, or analysis (Goodman et al., 2016).”
BackgroundInfluentialA model-based approach to assess reproducibility for large-scale high-throughput MRI-based studies
Jiao, Z., Lai, Y., Kang, J., Gong, W., Ma, L., Jia, T., Xie, C., Xiang, S., Cheng, W., Heinz, A., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119166 · Cited by 7
Fanelli, D. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0004 · Cited by 7
Self-citationHarremoës, P. (2019) · Publications · doi:10.3390/publications7030053 · Cited by 7
Zechner, D., Schulz, B., Tang, G., Abdelrahman, A., Kumstel, S., Seume, N., Palme, R., & Vollmar, B. (2022) · Animals · doi:10.3390/ani12212927 · Cited by 6
Reproducibility in Biomedical Research
Mullane, K., Curtis, M. J., & Williams, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00001-x · Cited by 11
A fully automated, transparent, reproducible, and blind protocol for sequential analyses
Beffara, B., Bret, A., & Nalborczyk, L. (2021) · Meta-Psychology · doi:10.15626/mp.2018.869 · Cited by 6
“To avoid these confusions, we adopt the terminology suggested by Goodman et al. (2016).”
MethodHow Theories of Induction Can Streamline Measurements of Scientific Performance
Perović, S., & Sikimić, V. (2019) · Journal for General Philosophy of Science · doi:10.1007/s10838-019-09468-4 · Cited by 6
“These negatively influence the replicability of biological experiments and slow down consensus (Goodman et al. 2016).”
BackgroundNassar, M., Auffan, M., Santaella, C., Masion, A., & Rose, J. (2021) · Frontiers in Environmental Science · doi:10.3389/fenvs.2021.625201 · Cited by 7
Reproducibility: does it really matter?Preprint
da Silva, J. A. T. (2016) · doi:10.15200/winn.146575.50444 · Cited by 6
Porta, M. S., & Bolúmar, F. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0181-3 · Cited by 7
Negrão, S., & Julkowska, M. (2020) · Wiley eBooks · doi:10.1002/9780470015902.a0028894 · Cited by 6
Shelchkova, N. D., Downey, J. E., Greenspon, C. M., Okorokova, E. V., Sobinov, A. R., Verbaarschot, C., He, Q., Sponheim, C., Tortolani, A. F., Moore, D. D., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.10.503543 · Cited by 6
Reproducibility and Data storage Checklist for Active Learning-Aided Systematic ReviewsPreprint
Lombaers, P., Bruin, J. D., & van de Schoot, R. (2023) · doi:10.31234/osf.io/g93zf · Cited by 6
Systematic review automation methods
Norman, C. R. (2020) · theses.fr (ABES) · doi:10.70675/044ffa34z5c9bz433az99c8z64697dcd28a2 · Cited by 8
A pragmatic guide to geoparsing evaluation
Gritta, M., Pilehvar, M. T., & Collier, N. (2018) · Language Resources and Evaluation · doi:10.1007/s10579-019-09475-3 · Cited by 95
The future of forecasting competitions: Design attributes and principles
Makridakis, S., Fry, C., Petropoulos, F., & Spiliotis, E. (2022) · Pure (University of Bath) · doi:10.1287/ijds.2021.0003 · Cited by 33
“This has been an increasing concern across sciences (Goodman et al., 2016), including the forecasting field (Boylan et al., 2015; Makridakis et al., 2018).”
BackgroundInvestigating the Heterogeneity of “Study Twins”
Röver, C., & Friede, T. (2024) · Biometrical Journal · doi:10.1002/bimj.202300387 · Cited by 10
“The degree to which it is possible to replicate the results of one study in another study is also referred to as “(results) reproducibility” (Goodman, Fanelli, and Ioannidis 2016; Plesser 2018).”
Digitalization of the natural sciences: Design science research and computational science
Storey, V. C., & Baskerville, R. (2024) · Decision Support Systems · doi:10.1016/j.dss.2024.114368 · Cited by 10
Formal definition and implementation of reproducibility tenets for computational workflows
Pritchard, N., & Wicenec, A. J. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107684 · Cited by 11
“Total replication is useful to certify a workflow as a ‘gold-standard’ [44] and is similar to the concept of results reproducibility [52].”
Voxel-Based Dose–Toxicity Modeling for Predicting Post-Radiotherapy Toxicity: A Critical Perspective
Puri, T. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14207248 · Cited by 5
Drude, N., Baselly, C., Gazda, M. A., May, J., Tienken, L., Abbasi, P., Weissgerber, T. L., & Burgess, S. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00188-0 · Cited by 5
Schlögl, M., Waltersdorfer, L., Regner, P., Siposova, A., & Brenning, A. (2026) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2026.106962 · Cited by 6
Validation of the phenol red thread test in a Chinese population
Hao, Y., Jin, T., Zhu, L., Zhao, M., Wang, S., Li, Z., Tian, L., & Jie, Y. (2023) · BMC Ophthalmology · doi:10.1186/s12886-023-03250-3 · Cited by 5
“Inter-examiner reproducibility is tested by two different examiners using the same raw materials or equipment to carry out the same procedure, and is used to evaluate the generalizability of the approach [32].”
MethodBianco, R., Speciani, M. C., Parpinel, M. T., Tesi, M., Ferraroni, M., & Edefonti, V. C. (2023) · Advances in Nutrition · doi:10.1016/j.advnut.2023.100165 · Cited by 5
“Within the movement supporting reproducible research in science [27], the current article builds on the fi rst 2 systematic reviews on reproducibility and validity of PCA/EFA-based DPs in nutritional epidemiology [7,10] and explores the cross-study reproducibility of PCA/EFA-based DPs in a case…”
MethodHankenson, F. C. (2024) · Journal of the American Association for Laboratory Animal Science · doi:10.30802/aalas-jaalas-24-000002 · Cited by 7
A framework for assessing the computational reproducibility of geo-simulation experiments
chen, M., Zhu, Z., Qian, Z., Ren, G., He, Y., Sun, L., Zhang, F., Yue, S., Lü, G., & Wen, Y. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106323 · Cited by 6
Moher, D., Boutron, I., Chan, A., Hopewell, S., Hróbjartsson, A., & Schulz, K. F. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111792 · Cited by 6
Ugolini, A., Cossellu, G., Rusconi, F., & Luca, S. D. (2021) · Australian Journal of Forensic Sciences · doi:10.1080/00450618.2021.1930157 · Cited by 5
Three cheers for descriptive statistics—and five more reasons why they matter
Credé, M., & Harms, P. D. (2021) · Industrial and Organizational Psychology · doi:10.1017/iop.2021.110 · Cited by 6
“At a time when issues of the reproducibility of research findings are of increasing concern (see Goodman et al., 2016; Grand et al., 2018), we believe that readers should not assume that data analyses have been correctly conducted or interpreted, should routinely consider whether the reported…”
BackgroundValidity, Reliability, and Significance: Empirical Methods for NLP and Data Science
Riezler, S., & Hagmann, M. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01137ed1v01y202110hlt055 · Cited by 32
Modelling efficient team structures in biology
Sikimić, V., & Herud-Sikimic, O. (2022) · Journal of Logic and Computation · doi:10.1093/logcom/exac021 · Cited by 6
Is Social Work Research in Crisis?
Dunleavy, D. J., & Lacasse, J. R. (2023) · Research on Social Work Practice · doi:10.1177/10497315231188684 · Cited by 6
“…fi ne reproducibility as: Using the same data and code, in an attempt to get the same results, and replicability as: Using the same methods, with new data, in an attempt to con fi rm the fi ndings of a prior study (see Goodman et al., 2016 for a differing taxonomy, which we are also sympathetic to).”
LISTER: Semiautomatic Metadata Extraction from Annotated Experiment Documentation in eLabFTW
Musyaffa, F. A., Rapp, K., & Gohlke, H. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00744 · Cited by 6
Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI
Moulton, R. H., McCully, G. A., & Hastings, J. D. (2024) · Computer Assisted Radiology and Surgery - International Congress and Exhibition · doi:10.1109/cars61786.2024.10778911 · Cited by 5
“However, the rapid evolution of software tools, libraries, and dependencies, coupled with the inherent complexity of these methodologies, has created a significant barrier to reproducing and validating published research results [8]–[11].”
Cueno, M. E., & Imai, K. (2017) · Experimental Gerontology · doi:10.1016/j.exger.2017.11.021 · Cited by 5
“Correctness of results obtained were confirmed through consilience in line with inferential reproducibility (Casadevall and Fang, 2010; Goodman et al., 2016).”
ResultReliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
Reproducibility by Other Means: Transparent Research Objects
McPhillips, T., Willis, C., Gryk, M. R., Núñez-Corrales, S., & Ludäscher, B. (2019) · International Conference on e-Science · doi:10.1109/escience.2019.00066 · Cited by 5
“[9], [17], [19], [23], [25], [26], [29], [31], [43], with respect to research results, processes, and settings is perhaps an even greater indication of both the significance of these efforts and the challenges they face.”
BackgroundResultReproducible topographic analysis
Grieve, S., Clubb, F. J., & Mudd, S. M. (2020) · Developments in earth surface processes · doi:10.1016/b978-0-444-64177-9.00012-6 · Cited by 6
Tebbe, J., Humble, E., Stoffel, M. A., Tewes, L. J., Müller, C., Forcada, J., Caspers, B. A., & Hoffman, J. I. (2020) · PeerJ · doi:10.7717/peerj.10131 · Cited by 6
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.