What does research reproducibility mean?
The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 901–1,000 of 1,003
Bertoldo, G. (2019) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/w63h7 · Cited by 2
Establishing a reproducible and sustainable analysis workflowPreprint
Ramduny, J., Garcia, M., & Kelly, C. A. M. (2023) · doi:10.31219/osf.io/rcxg8 · Cited by 2
Psychiatric Neuroimaging Designs for Individualised, Cohort, and Population StudiesPreprint
Gell, M., Noble, S., Laumann, T. O., Nelson, S. M., & Tervo‐Clemmens, B. (2024) · doi:10.31234/osf.io/278ef · Cited by 2
A New Standard for the Analysis and Design of Replication Studies
Held, L. (2019) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12493 · Cited by 83
“Recently the lack of a single accepted definition of replicability has been emphasized by Goodman et al. (2016) who called for a better understanding of the relationship between reproducibility and the truth of scientific claims.”
Teaching reproducible research for medical students and postgraduate pharmaceutical scientists
Meid, A. D. (2021) · BMC Research Notes · doi:10.1186/s13104-021-05862-8 · Cited by 2
“Following the “new lexicon for research reproducibility” [6], (methods) reproducibility is based on the same research conditions and is a fundamental requirement for successful replication (results reproduction).”
MethodVovk, V. (2021) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12656 · Cited by 1
Should I disclose my dataset? Caveats between reproducibility and individual data rights
Benatti, R. M., Villarroel, C. M. L., Avila, S., Colombini, E. L., & Severi, F. C. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.20 · Cited by 1
The epistemic status of reproducibility in political fact-checking
Fernández-Roldán, A., & Teira, D. (2024) · European Journal for Philosophy of Science · doi:10.1007/s13194-024-00575-8 · Cited by 2
Aldon-Villegas, R., Chamorro-Moriana, G., Espuny-Ruiz, F., & Benítez-Lugo, M.-L. (2024) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-024-08075-9 · Cited by 1
eCommerce Search and Discovery
Kallumadi, S. T., & King, T. H. (2024) · ACM eBooks · doi:10.1145/3674127.3674140 · Cited by 1
Christ, A., Birnbaum, L., & Kröner, S. (2025) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-025-01301-7 · Cited by 1
“Autorinnen und Autoren von empirischen Arbeiten können Reproduktionen er-leichtern, indem sie in dafür geeigneten Repositorien sowie direkt in ihren Arbeiten die dafür erforderlichen Informationen berichten bzw. zur Verfügung stellen (Al-lison et al. 2018; Barba 2018; Goodman et al. 2016).”
Meirmans, S., Luijken, K., Bruijning-Verhagen, P. C. J. L., Groenwold, R. H. H., & de Boer, A. R. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111785 · Cited by 2
The Observed Availability of Data and Code in Earth Science and Artificial Intelligence
Jones, E. A., McClung, B., Fawad, H., & McGovern, A. (2025) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-24-0147.1 · Cited by 1
A Quantitative Metric for Reproducibility in Human Robot Interaction Experiments
Tavares, B. T., Freire, E., Molina, L., Carvalho, J. G. N., & Carvalho, E. (2025) · 2025 Brazilian Conference on Robotics (CROS) · doi:10.1109/cros66186.2025.11066103 · Cited by 1
“Different types of replication, such as conceptual replication [5], allow for modifications to the original method to explore under what conditions and for whom a finding holds true.”
InfluentialMisleading biases in methods for estimating wolf abundance using spatial models
Crabtree, R. L., Conner, M. M., & Treves, A. (2025) · Academia Biology · doi:10.20935/acadbiol7924 · Cited by 1
“Before a specific finding can be replicated and confirmed by qualified investigators following every step of the original methods, re-producibility demands two tests of the original findings [98–100].”
Donat-Bouillud, P., Křikava, F., Krynski, S., & Vítek, J. (2025) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3736731.3746156 · Cited by 1
“Note that we do not aim at achieving result reproducibility [13] nor replicability [1], where the goal is to find the same results but not necessarily using the same exact computational steps.”
Bramlett, A. A., & Wiener, S. (2025) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2025.100275 · Cited by 2
Correia, L., Goos, J.-C., Bäck, T., & Kononova, A. V. (2025) · Big Data Research · doi:10.1016/j.bdr.2025.100573 · Cited by 3
“In case future work aims to replicate the results of this paper, we encourage deviating from the experimental setup outlined in this paper [16], though, as Bartz-Beielstein et al. [4] point out, there is no definition for how different an experimental setup needs to be for results to be considered…”
Adrian, Q., Cohen, R. F., Plasse, J., Chéreau, I., Gouache, B., Legros-Lafarge, E., Massoubre, C., Guillard-Bouhet, N., Temblay, L., Franck, N., et al. (2025) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2025.152651 · Cited by 2
Reproducibility in the Control of Autonomous Mobility-on-Demand Systems
Li, X., Alharbi, M. S. M., Gammelli, D., Harrison, J., Rodrigues, F., Schiffer, M., Pavone, M., Frazzoli, E., Zhao, J., & Zardini, G. (2026) · IEEE Transactions on Robotics · doi:10.1109/tro.2026.3666153 · Cited by 5
“The term reproducibility has been used to describe varying degrees of rigor in how scientific findings are communicated and validated [11], [12].”
Bordignon, A., Collina, F., Fabbri, F., Fanini, B., Ferdani, D., Rega, M. F., & Sullini, M. (2026) · Peer Community Journal · doi:10.24072/pcjournal.696 · Cited by 1
Supporting reanalysis and reuse of clinical trial data: a case study
Burgwinkel, C., Chiam, H. C., Tai, K., Wang, J., Ali, M. H., Fallah, S. S., Matbouriahi, M., Obinwanne, T., Papapostolou, G., Riedha, M., et al. (2026) · Trials · doi:10.1186/s13063-026-09646-y · Cited by 2
“Together, these developments underline that transparent reporting is not only a cornerstone of open science but also an essential foundation for reproducibility, without which the validity of research findings cannot be reliably assessed [7, 9].”
Cacciato‐Salcedo, S., Jaokar, S., Ingham, N. J., Malmierca, M. S., & Petrinovic, M. M. (2026) · Autism Research · doi:10.1002/aur.70275 · Cited by 1
“Addressing this will require collaborative, multi-site initiatives with sufficiently powered samples, preregistered analysis plans, and transparent reporting of effect sizes and confidence intervals (Button et al. 2013; Goodman et al. 2016; Ioannidis 2005).”
Data and code (Un)availability in sports meta-analysis studies
Wolff, P. A., Sainani, K. L., & Borg, D. N. (2026) · Annals of Epidemiology · doi:10.1016/j.annepidem.2026.110126 · Cited by 1
Sirago, G., Solarino, B., Dell'Erba, A., & Ferorelli, D. (2026) · International Journal of Legal Medicine · doi:10.1007/s00414-026-03849-3 · Cited by 1
““Minimum” should not be read as permissive: in evidence contexts, minimum is the threshold for falsifiability [ 76 ].”
Irreproducible research and a typology of replication efforts
Treves, A., Chinn, S., & Krofel, M. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00203-4 · Cited by 2
“Replication is an attempt to repeat a finding and is therefore a benchmark of reproducibility [5, 6].”
Pejdo, D., Buljan, I., & Marušić, A. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00217-y · Cited by 1
“Despite increasing awareness, however, these efforts to enhance research reproducibility remain fragmented, and the implementation of evidence-based practices is inconsistent across research domains [10, 11].”
Toward standardization in arthropod and biodiversity genome projects
Tvedte, E. S., Bucher, G., Luecke, D. M., Molik, D. C., Sylvester, T. P., Blaxter, M., Elsik, C. G., Howe, K., McKenna, D. D., Murphy, T. D., et al. (2026) · Genetics · doi:10.1093/genetics/iyag172 · Cited by 1
Lee, K., Kumagai, C., Komuro, M., Miyo, K., & Hoshimoto, H. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02447-5 · Cited by 1
“Therefore, wherever feasible, it is recommended that researchers and data admin - istrators record not only the study period and extraction con - ditions but also the extraction date and dataset version used for analysis [ 20 , 21].”
Enhancing reproducibility in hybrid Earth system models
Chen, M., Zhu, Z., Wagener, T., Boers, N., Müller, R. D., Strobl, J., Camps‐Valls, G., Batty, M., Jakeman, A. J., Kolditz, O., et al. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01299-5 · Cited by 2
“It allows research to be interrogated, validated and built upon, transforming individual findings into shared knowledge 9,11 .”
InfluentialReproducible Pattern Recognition Research: The Case of Optimistic SSL
Krijthe, J. H., & Loog, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56414-2_4 · Cited by 1
“We covered our approach to reproducing our paper on optimistic semi-supervised learning and showed some additional interesting, and nontrivial results by making slight adjustments to the figures and experiments which the reproducible nature of the paper allows.”
InfluentialAn Introduction to Systematic Reviews and Meta-Analyses for Exotic Animal Practitioners
Reynders, R. M. (2017) · Veterinary Clinics of North America Exotic Animal Practice · doi:10.1016/j.cvex.2017.04.005 · Cited by 1
Progress in Biomedical Sciences and Raw Data: Ethical Dilemmas
Casacuberta, D., & Tassani, S. (2019) · Springer eBooks · doi:10.1007/978-3-030-28626-2_6 · Cited by 1
Reproducibility: Buy Low, Sell High
Maxion, R. A. (2020) · IEEE Security & Privacy · doi:10.1109/msec.2020.3005077 · Cited by 1
“Goodman et al. 6 argue persuasively that none of these terms is well defined, and even when fulfilled they don’t satisfy the higher goal of determining truth.”
BackgroundAn Assessment of Scoping Review Reporting within Paramedicine: A Scoping Review
Williams, B. A., & Beovich, B. (2020) · Australasian Journal of Paramedicine · doi:10.33151/ajp.17.834 · Cited by 2
“Transparent, comprehensive reporting of research syntheses are valuable for understanding, interpretation and to enable reproduction of a study’s methods by other researchers (38).”
BackgroundIrreproducibility –The deadly sin of preclinical research in drug development
Pillai, S., Kobayashi, K., Michael, M., & Meena, A. (2020) · Journal of Pre-Clinical and Clinical Research · doi:10.26444/jpccr/131017 · Cited by 1
“Irreproducibility of research findings has become a serious concern to funders and policy makers [4].”
BackgroundStudy Registration for the Field of Prevention Science: Considering Options and Paths Forward
Spybrook, J. K., Maynard, R. A., & Anderson, D. (2021) · Prevention Science · doi:10.1007/s11121-021-01290-z · Cited by 4
Introduction: Methods for Collecting EEG Data for Frequency Analyses in Humans
Gable, P. A., Miller, M. W., & Bernat, E. M. (2022) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780192898340.013.1 · Cited by 2
The role of Infrastructure for Software in Open Science
Lucke, U. (2022) · doi:10.4000/books.oep.16184 · Cited by 1
Actes des Journées européennes de la science ouverte
Conference, O. S. E. (2022) · doi:10.4000/books.oep.15822 · Cited by 2
Gabriel, M. (2022) · doi:10.4000/books.oep.16089 · Cited by 1
Võikar, V., Casarotto, P. C., Glerean, E., Laakso, K., Saurio, K. J., Karhulahti, V., & Scherer, A. (2023) · Journal for Reproducibility in Neuroscience · doi:10.36850/jrn.2023.e47 · Cited by 1
Data Framework Design of EDISON 2.0 Digital Platform for Convergence Research
Han, S., Lee, J., Jeon, I., Lee, J.-C., & Choi, H. (2023) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2023.08.019 · Cited by 4
“ EDISON 2.0 should be able to support the entire end-to-end research process; this is related to research reproducibility [15,16].”
BackgroundInfluentialEvolution of fertilization ability in obligatorily outcrossing populations of Caenorhabditis elegans
Palka, J. K., Dyba, A., Brzozowska, J., Antoł, W., Sychta, K., & Prokop, Z. M. (2023) · PeerJ · doi:10.7717/peerj.15825 · Cited by 1
Digital Twin-Based Concept for Reliable Research Data Management
Rache, A., Häußermann, T. M., Lehmann, J., & Reichwald, J. (2023) · Proceedings of the Conference on Research Data Infrastructure · doi:10.52825/cordi.v1i.297 · Cited by 1
The replication crisis in science and its overlap into kinantropology: an introduction
Baloun, L., & Mališ, J. (2023) · Tělesná kultura · doi:10.5507/tk.2023.002 · Cited by 1
How to Address Publication Overload in Environmental Science
Brandt, W. T., & Tague, C. (2023) · Eos · doi:10.1029/2023eo230361 · Cited by 1
Subramaniyan, M., Wang, C., Laxminarayan, S., Vital-Lopez, F. G., Hughes, J., Doty, T. J., & Reifman, J. (2023) · Journal of Sleep Research · doi:10.1111/jsr.14060 · Cited by 1
Towards more reproducibility in vegetation research
Sperandii, M. G., Bazzichetto, M., Mendieta‐Leiva, G., Schmidtlein, S., Bott, M., de Lima, R. A. F., Pillar, V. D., Price, J. N., Wagner, V., & Chytrý, M. (2024) · Journal of Vegetation Science · doi:10.1111/jvs.13224 · Cited by 2
Relationship between physical exercise load and return-to-work in female breast cancer survivors
Gilabert, H. C., Figueras, J. A., Matas, S., Anzano, A., & Mas-Alòs, S. (2024) · Physical Activity Exercise and Cancer · doi:10.61486/myeo2515 · Cited by 1
Population neuroscience data sharing: Policy ecology, infrastructure, practices and challenges
Jiang, L., Gao, P., & Zhou, Y. (2024) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/tb-2024-0147 · Cited by 2
Predicting the Sentiment of Review Aspects in the Peer Review Text using Machine Learning
Basuki, S., Sari, Z., Tsuchiya, M., & Indrabayu, R. (2024) · Kinetik Game Technology Information System Computer Network Computing Electronics and Control · doi:10.22219/kinetik.v9i4.2042 · Cited by 2
Prerequisite for reproducible science: a call to embrace code sharing
Gun, A., & Garg, T. (2024) · The Lancet Regional Health - Southeast Asia · doi:10.1016/j.lansea.2024.100472 · Cited by 1
Phenomenological Modeling of Diverse and Heterogeneous Synaptic Dynamics at Natural Density
Korcsak-Gorzo, A., Linssen, C., Albers, J., Dasbach, S., Duarte, R., Kunkel, S., Morrison, A., Senk, J., Stapmanns, J., Tetzlaff, T., et al. (2024) · Neuromethods · doi:10.1007/978-1-0716-4019-7_15 · Cited by 1
“Similar to the model descriptions, it increases the reproducibility of methods and results (Goodman et al., 2016) to use and contribute to existing simulation frameworks by reporting bugs, improving implemented methods, and developing and publishing custom modules of the respective framework, e.”
MethodAttribute Relevance Score: A Novel Measure for Identifying Attribute Importance
Neirz, P., Allende, H., & Saavedra, C. (2024) · Algorithms · doi:10.3390/a17110518 · Cited by 4
“Goodman, Fanelli, and Ioannidis [15] categorize reproducibility into three essential types: methods reproducibility, which ensures that technical procedures can be precisely replicated; results reproducibility, which guarantees that reimplementations of a method produce statistically consistent…”
Galuščáková, P., Oard, D. W., & Nair, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674136 · Cited by 3
Gómez, E., Cuesta, H., Gkiokas, A., Gómez-Cañón, J. S., Porcaro, L., & Yesiler, F. (2024) · ACM eBooks · doi:10.1145/3674127.3702959 · Cited by 5
Baeza-Yates, R. A., Murgai, L., & Dudy, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674138 · Cited by 1
Williams, R., Jenkins, D. A., Bolton, T., Heald, A. H., Mizani, M. A., Sperrin, M., & Peek, N. B. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-093080 · Cited by 1
Peng, Y., Lyu, P., & Peng, X. (2025) · Cancer Pathogenesis and Therapy · doi:10.1016/j.cpt.2025.07.002 · Cited by 6
“19 Given the persistent and critical reporting gaps identified in oncology trials, coupled with the field's unique methodological and practical complexities (e.g., biomarker stratification, combination therapies with potential interactions, complex interventions such as cellular therapies, diverse…”
Vardalaki, D., Pham, T. L. D., Frosch, M. P., Cosgrove, G. R., Carter, B. S., Richardson, R. M., Cash, S. S., & Harnett, M. T. (2025) · Scientific Reports · doi:10.1038/s41598-025-18207-3 · Cited by 3
Tang, B. L. (2018) · Accountability in Research · doi:10.1080/08989621.2018.1479257 · Cited by 1
““Reproducibility in research” has a variety of interpretations, and these have recently been categorized to include methods reproducibility, results reproducibility, and inferential reproducibility (Goodman, Fanelli, and Ioannidis 2016).”
MethodIssues in preclinical radiopharmaceutical research: Significance, relevance and reproducibility
Kilbourn, M. R., & Scott, P. J. H. (2018) · Nuclear Medicine and Biology · doi:10.1016/j.nucmedbio.2018.07.002 · Cited by 1
Presidential address: Six open questions to genetic epidemiologists
König, I. R. (2019) · Genetic Epidemiology · doi:10.1002/gepi.22191 · Cited by 1
Riedl, V., Agatz, A., Benstead, R., & Ashauer, R. (2019) · Environmental Toxicology and Chemistry · doi:10.1002/etc.4393 · Cited by 1
“Variability has differing definitions in the literature (Organisation for Economic Co‐operation and Development 2005; Goodman et al. 2016; Leek and Jager 2017; Riedl et al. 2018); in the present study we describe differences within experiments as intratest variability, differences between…”
BackgroundMethodological Transparency and Big Data: A Critical Comparative Analysis of Institutionalization
Sanfilippo, M. R., & McCoy, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15742-5_5 · Cited by 1
Faggion, C. M., & Hagenfeld, D. (2019) · Journal of Dentistry · doi:10.1016/j.jdent.2019.05.029 · Cited by 1
OpenAIRE: Advancing Open Science
Manghi, P., Artini, M., Atzori, C., Baglioni, M., Bardi, A., Dimitropoulos, H., Foufoulas, I., Iatropoulou, K., Manola, N., & Príncipe, P. (2018) · ISTI Open Portal · doi:10.26069/greynet-2018-000.010-gg · Cited by 1
Fehrmann, P. G., & Mamolen, M. (2020) · IASSIST Quarterly · doi:10.29173/iq968 · Cited by 1
Massiani, J. (2020) · BOA (University of Milano-Bicocca) · doi:10.6092/issn.2036-5195/10719 · Cited by 1
Vilgis, T. A. (2020) · Springer eBooks · doi:10.1007/978-3-662-61151-7_6 · Cited by 1
Held, L., & Pawel, S. (2020) · Statistics in Biopharmaceutical Research · doi:10.1080/19466315.2020.1828161 · Cited by 2
Pascual, G. D. (2021) · Law and Method · doi:10.5553/rem/.000053 · Cited by 1
The Reproducibility Crisis in Science Education Studies: Causes and Solutions
Nakamura, D., HARADA, Y., HISASAKA, T., UNZAI, H., & Matsuura, T. (2021) · Journal of Research in Science Education · doi:10.11639/sjst.sp20016 · Cited by 5
An Introduction to Reproducibility in the Context of Animal Research
Sánchez-Morgado, J. M., & Brønstad, A. (2021) · Laboratory animal science and medicine · doi:10.1007/978-3-030-66147-2_1 · Cited by 1
Defining the Intervention Logic of Access-To-Data Measures: A Problem Analysis
Kim, D. (2021) · Munich studies on innovation and competition · doi:10.1007/978-3-030-86778-2_6 · Cited by 1
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 2
Cannabinoid-Based Medicines and Cancer Symptom Management
Daeninck, P. J., Rudney, R., & Davis, M. P. (2022) · Springer eBooks · doi:10.1007/978-3-030-89918-9_5 · Cited by 1
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 2
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 2
The use of the phrase “data not shown” in dental research
Raittio, E. J., Sofi‐Mahmudi, A., & Shamsoddin, E. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272695 · Cited by 2
“The foundation of science and research is sustainable, valid and reliable when the results are available to be tested, replicated and reproduced [1–4].”
Sharing of Data from Clinical Research Projects: Guidance from the SCTO’s CTU Network
Gahl, B., Haynes, A. G., Sluka, C., Dupuis‐Lozeron, E., Jörger, F., Schur, R., Christen, A., & Trelle, S. (2021) · doi:10.54920/scto.2021.02 · Cited by 1
Vilgis, T. A. (2022) · Springer eBooks · doi:10.1007/978-3-662-65108-7_6 · Cited by 1
Open Science, Publications and Code
Harrison, M. (2022) · doi:10.4000/books.oep.16187 · Cited by 1
Rethinking Psychological Research Methods From The Seven Deadly Sins of Psychology
KUDO, Y., HAEBARA, T., Murai, J., Okada, K., Kunisato, Y., HIRAISHI, K., & SHIBAYAMA, T. (2022) · The Annual Report of Educational Psychology in Japan · doi:10.5926/arepj.61.291 · Cited by 1
“…究にとっても無縁ではありえないだろう。本チュート リアル・セミナーでは,3人の話題提供者が,上記の 7つの罪のうちのいくつかと絡めつつ,各々の専門を 反映させながら実体験に基づいた(同書でいうところの) 「救済」に関する発表をしていく。以上を通し,日頃の 研究手続きの中に潜む落とし穴について学び「建前と しての仮説演繹法」から脱却を図るために何をすべき か,考える機会としたい。 話題提供 有意差が出るまでデータを集めてもよいですか―任意 停止問題とベイズファクター 岡田謙介 任意停止問題 従来主流であった頻度論の統計学において,サンプ…”
Building the Software Pillar of Open Science
Cosmo, R. D. (2022) · doi:10.4000/books.oep.16182 · Cited by 3
(México), U. N. A. D. M., Canet, C., Alvarado-Sizzo, I., (México), U. N. A. D. M., Mora-Chaparro, J. C., (México), U. N. A. D. M., Cruz-Pérez, M. Á., (México), U. L. S. P., Salgado-Martínez, E., (México), U. N. A. D. M., et al. (2023) · PASOS Revista de Turismo y Patrimonio Cultural · doi:10.25145/j.pasos.2023.21.045 · Cited by 1
Scikick: A sidekick for workflow clarity and reproducibility during extensive data analysis
Carlucci, M., Bareikis, T., Koncevičius, K., Gibas, P., Kriščiūnas, A., Petronis, A., & Oh, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289171 · Cited by 1
“Research reproducibility, in the many forms it takes [1–3], is essential to the scientific method.”
Leung, T. H., Ho, J., Wang, X., Lam, W., & Pang, H. (2023) · Journal of Clinical and Translational Science · doi:10.1017/cts.2023.699
A mathematical theory of knowledge, science, bias and pseudosciencePreprint
Fanelli, D. (2016) · doi:10.7287/peerj.preprints.1968v1 · Cited by 1
Self-citationThe Solution to Science's Replication CrisisPreprint
Knuteson, B. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2835131 · Cited by 2
“[47] Goodman SN, Fanelli D, Ioannidis JP.”
Commentary: The challenge of evaluating and improving evidence when research lacks reproducibility
Wallach, J. D., & Basu, S. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw338 · Cited by 1
“The term ‘reproducible’ has been used in varied contexts over the years; here, we follow Goodman et al.2 to refer to three key types of reproducibility: (i) ‘methods reproducibility’, which indicates that a study has provided enough information about the experimental or computation procedures so that subsequent author…”
BackgroundMethodInfluentialOrtmann, A. (2017) · Journal of Economic Psychology · doi:10.1016/j.joep.2017.02.010 · Cited by 1
Bad Smells in Software Analytics PapersPreprint
Krishna, R., Majumder, S., Menzies, T., & Shepperd, M. (2018) · arXiv (Cornell University) · Cited by 2
Exploration of Reproducibility Issues in Scientometric Research Part 2: Conceptual ReproducibilityPreprint
Velden, T., Hinze, S., Scharnhorst, A., Schneider, J. W., & Waltman, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05026 · Cited by 5
“The diversity of perspectives has led to a thorough confusion of terminology around reproducibility, including antithetical definitions of the terms replicability and reproducibility (Goodman et al. 2016; Barba 2018).”
Reply to letter to the editor by C. Faggion: reproducibility and reporting guidelines
Altman, D. G., & Moher, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.04.007 · Cited by 1
A human embryonic stem cell model of Aβ-dependent chronic progressive neurodegenerationPreprint
Ubina, T., Magallanes, M., Srivastava, S., Warden, C., Yee, J., & Salvaterra, P. M. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/419978 · Cited by 1
Multi-stage models for the failure of complex systems, cascading disasters, and the onset of diseasePreprint
Webster, A. J. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/476242 · Cited by 1
Evaluation of Reproducible and Transparent Research Practices in Pulmonology PublicationsPreprint
Smith, C. A., Nolan, J., Tritz, D. J., Heavener, T. E., Pelton, J., Cook, K., & Vassar, M. B. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/726505 · Cited by 1
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.