What does research reproducibility mean?
The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 301–400 of 1,003
Weller, D. L., Kovač, J., Kent, D. J., Roof, S. E., Tokman, J. I., Mudrak, E. L., Kowalcyk, B. B., Oryang, D., de Aceituno, A. M. F., & Wiedmann, M. (2017) · Food Microbiology · doi:10.1016/j.fm.2017.06.009 · Cited by 30
Gorman, D. M. (2017) · Children and Youth Services Review · doi:10.1016/j.childyouth.2017.07.009 · Cited by 33
Strutz, P., Kronzer, V. L., Tzeng, W. S., Arrington, B., McKinnon, S., Abdallah, A. B., Haroutounian, S., & Avidan, M. S. (2019) · Anaesthesia · doi:10.1111/anae.14855 · Cited by 32
“It is also important to note that hypothesisgenerating findings are often not reproduced in larger, more rigorous studies [44, 45].”
BackgroundApparent size and morphology of bacterial microcompartments varies with technique
Kennedy, N. W., Hershewe, J. M., Nichols, T. M., Roth, E. W., Wilke, C., Mills, C. E., Jewett, M. C., & Tullman‐Ercek, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0226395 · Cited by 31
“The issue has seeped into fields across every domain of scientific inquiry [2].”
BackgroundThe critical need to foster computational reproducibility
Reinecke, R., Trautmann, T., Wagener, T., & Schüler, K. (2022) · Environmental Research Letters · doi:10.1088/1748-9326/ac5cf8 · Cited by 33
“…et al (2016) in their assessment that without being able to redo the exact experiment ( methods reproducibility Goodman et al (2016)) we cannot advance our science because we will not be able to assess which assumptions may have led to the outcomes ( results reproducibility Goodman et al (2016)).”
BackgroundMethodChoi, S., Hong, S. P., Bae, J. H., Suh, S. H., Bae, H., Kang, K. P., Lee, H. J., & Koh, G. Y. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000075 · Cited by 35
Workflows for microscopy image analysis and cellular phenotyping
Wollmann, T., Erfle, H., Eils, R., Rohr, K., & Gunkel, M. (2017) · Journal of Biotechnology · doi:10.1016/j.jbiotec.2017.07.019 · Cited by 32
“There exists a vast variety of microscopy image analysis software with multiple versions, which often impedes reproducing analysis results (Goodman et al., 2016), or sharing workflows.”
BackgroundReproducibility and replicability of software defect prediction studies
Mahmood, Z., Bowes, D., Hall, T., Lane, P. C. R., & Petrić, J. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.02.003 · Cited by 35
“Goodman et al. [16] suggest that in any scientific field the kind of replication must be clearly specified.”
BackgroundReporting guidelines for oncology research: helping to maximise the impact of your research
MacCarthy, A., Kirtley, S., de Beyer, J. A., Altman, D. G., & Simera, I. (2018) · British Journal of Cancer · doi:10.1038/bjc.2017.407 · Cited by 30
“Further, other researchers may be prevented from repeating the study (Goodman et al, 2016).”
BackgroundCentralized scientific communities are less likely to generate replicable results
Danchev, V., Rzhetsky, A., & Evans, J. A. (2019) · eLife · doi:10.7554/elife.43094 · Cited by 27
The Search for Clinically Useful Biomarkers of Complex Disease: A Data Analysis Perspective
Considine, E. C. (2019) · Metabolites · doi:10.3390/metabo9070126 · Cited by 30
Estimating the reproducibility of social learning research published between 1955 and 2018
Minocher, R., Atmaca, S., Bavero, C., McElreath, R. L., & Beheim, B. (2021) · Royal Society Open Science · doi:10.1098/rsos.210450 · Cited by 48
Repeatable high-resolution statistical downscaling through deep learning
Quesada‐Chacón, D., Barfus, K., & Bernhofer, C. (2022) · Geoscientific model development · doi:10.5194/gmd-15-7353-2022 · Cited by 40
“…and this implies to provide “enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated” (Goodman et al., 2016 Due to its less conflictive definition and what is achievable within a publication, repeatability will be pursued hereafter.”
BackgroundMethodInfluentialMosnier, A., Schmidt‐Traub, G., Obersteiner, M. H., Jones, S. K., Javalera-Rincon, V., DeClerck, F., Thomson, M. J., Sperling, F., Harrison, P. A., Pérez-Guzmán, K., et al. (2022) · Sustainability Science · doi:10.1007/s11625-022-01227-7 · Cited by 38
“It is important to ensure that the analysis can be scrutinised and repeated by other persons than the authors (Goodman et al. 2016; Wilkinson et al. 2016).”
BackgroundPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
Methodological challenges in studying the COVID-19 pandemic crisis
Knottnerus, J. A., & Tugwell, P. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.04.001 · Cited by 31
“[4] propose the term of ‘inferential reproducibility’ for making of knowledge claims of similar strength from either a study replication or reanalysis of original data.”
BackgroundDudda, L., Kormann, E., Kozula, M., DeVito, N. J., Klebel, T., Dewi, A. P. M., Spijker, R., Stegeman, I., Van den Eynden, V., Ross-Hellauer, T., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242057 · Cited by 28
“[10] call ‘methods reproducibility’).”
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28
OpenSAFELY: A platform for analysing electronic health records designed for reproducible research
Nab, L., Schaffer, A. L., Hulme, W., DeVito, N. J., Dillingham, I., Wiedemann, M., Andrews, C. D., Curtis, H., Fisher, L., Green, A., et al. (2024) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5815 · Cited by 34
“11 Closely related to the Turing Way definition, Goodman et al. 7 defined methods reproducibility as: ‘ the ability to implement, as exactly as possible, the experimental and computational procedures, with the same data and tools , to obtain the same results ’ (definition 3).”
Arroyo-Araujo, M., Graf, R., Maco, M., van Dam, E., Schenker, E., Drinkenburg, W. H. I. M., Koopmans, B., de Boer, S. F., Cullum‐Doyle, M., Noldus, L. P. J. J., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-47981-0 · Cited by 34
Statistical significance gives bias a free pass
Amrhein, V., Greenland, S., & McShane, B. B. (2019) · European Journal of Clinical Investigation · doi:10.1111/eci.13176 · Cited by 31
Reito, A. R., Raittio, L., & Helminen, O. (2020) · JBJS Reviews · doi:10.2106/jbjs.rvw.19.00079 · Cited by 35
Downey, J. E., Quick, K. M., Schwed, N., Weiss, J. M., Wittenberg, G. F., Boninger, M. L., & Collinger, J. L. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa120 · Cited by 31
Evaluating LC-HRMS metabolomics data processing software using FAIR principles for research software
Du, X., Dastmalchi, F., Ye, H., Garrett, T. J., Diller, M. A., Liu, M., Hogan, W. R., Brochhausen, M., & Lemas, D. J. (2023) · Metabolomics · doi:10.1007/s11306-023-01974-3 · Cited by 27
“…Del Carratore et al., 2019 ; Dührkop et al., 2019 ; Fischer et al., 2022 ; Franceschi et al., 2014 ; Gatto et al., 2021 ; Giacomoni et al., 2015 ; Guo et al., 2021 ; Helmus et al., 2021 ; Huan & Li, 2015a, b ; Huang et al., 2014 ; F. Huber, Ridder, et al., 2021 ; Hughes et al., 2014 ; Jaitly et…”
Tadmor, D., Till, K., Phillips, G., Brown, J. C., Fairbank, L., Hendricks, S., Johnston, R. D., Longworth, T., Stokes, K., & Jones, B. L. (2023) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2023.09.015 · Cited by 28
“19,20 Therefore, the aim of this study was to determine the rates of concussion underreporting in SL players and establish why players may not report symptoms or suspected concussions, alongside establishing basic concussion knowledge and establishing awareness of the long-term implications of…”
Multilayer network modeling of integrated biological systems
Domenico, M. D. (2017) · Physics of Life Reviews · doi:10.1016/j.plrev.2017.12.006 · Cited by 31
“Despite the success of multilayer network modeling and analysis in systems biology and systems medicine, some methodological challenges are still to be tackled to build consistent, replicable and reproducible [34, 35, 36, 37] representations of multi-omics, connectomics and intercellular interactions as the ones centr…”
BackgroundWyndham, J. M., Weisenberg, N., McCarty, C. A., Goldman, G. T., Brown, M., & Borenstein, J. (2022) · doi:10.1126/aaas.ade8203 · Cited by 32
A New Account of Replication in the Experimental Life Sciences
Guttinger, S. (2019) · Philosophy of Science · doi:10.1086/703555 · Cited by 27
“There is a broad consensus, also among scientists, that replications are rarely performed in the experimental sciences (Collins 1985; Baker 2016b; Goodman, Fanelli, and Ioannidis 2016).4 There are few empirical data on the exact extent of the problem, but current estimates place the percentage of…”
BackgroundChallenges and Opportunities in dMRI Data Harmonization
Zhu, A. H., Moyer, D., Nir, T. M., Thompson, P. M., & Jahanshad, N. (2019) · Mathematics and visualization · doi:10.1007/978-3-030-05831-9_13 · Cited by 33
Mische, S. M., Fisher, N. C., Meyn, S. M., Sol‐Church, K., Hegstad‐Davies, R. L., Weis‐Garcia, F., Adams, M., Ashton, J. M., Delventhal, K. M., Dragon, J. A., et al. (2020) · Journal of Biomolecular Techniques JBT · doi:10.7171/jbt.20-3101-003 · Cited by 34
Standards of reporting: the use of CONSORT PRO and CERT in individuals living with osteoporosis
Mack, D. E., Wilson, P. M., Santos, E., & Brooks, K. (2017) · Osteoporosis International · doi:10.1007/s00198-017-4249-z · Cited by 29
“Concerns about the reproducibility of scientific research have been steadily increasing in recent years [1, 2].”
BackgroundThe Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 32
“(Goodman et al., 2016, p.2) 4 researchers’ incentives and their possible responses to reforms in scientific practices.”
Badges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.2 · Cited by 26
Standards for design and measurement would make clinical research reproducible and usable
Dickersin, K., & Mayo‐Wilson, E. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708273114 · Cited by 27
Measurement accuracy and uncertainty in plant biomechanics
Nelson, N., Stubbs, C. J., Larson, R., & Cook, D. D. (2019) · Journal of Experimental Botany · doi:10.1093/jxb/erz279 · Cited by 26
What smells? Gauging attention to olfaction in canine cognition research
Horowitz, A., & Franks, B. (2019) · Animal Cognition · doi:10.1007/s10071-019-01311-z · Cited by 28
“Reliable reporting of SMELL sources would represent the kind of methodological hygiene necessary for the kind of repetition and replication of research that is the backbone of science (Goodman et al. 2016).”
BackgroundClinical Application of Computational Methods in Precision Oncology: A Review
Panagiotou, O. A., Hoffman-Hōgg, L., Hricak, H., Khleif, S. N., Levy, M. A., Magnus, D. C., Murphy, M. J., Patel, B., Winn, R. A., Nass, S. J., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.1247 · Cited by 31
An evaluation of the practice of transparency and reproducibility in addiction medicine literature.
Adewumi, M. T., Vo, N., Tritz, D. J., Beaman, J., & Vassar, M. (2020) · Addictive Behaviours · doi:10.1016/j.addbeh.2020.106560 · Cited by 22
Promoting Open Science Through Research Data Management
Borghi, J., & Gulick, A. E. V. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.9497f68e · Cited by 40
van Zwet, E. W., & Goodman, S. N. (2022) · Statistics in Medicine · doi:10.1002/sim.9406 · Cited by 31
Self-citationNahum-Shani, I. B., Wetter, D. W., & Murphy, S. A. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-90045-4.00012-5 · Cited by 30
Mikhael, S. S., Hoogendoorn, C., Hernández, M. V., & Pernet, C. R. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.02.082 · Cited by 29
“These underlying differences, more than differences in parcels themselves which can be combined, are likely at the root of variable, possibly irreplicable, results (i.e. non-equivalence of effect sizes (Goodman et al., 2016)), obtained with different software.”
Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology
Sauter, D., Lodde, G., Nensa, F., Schadendorf, D., Livingstone, E., & Kukuk, M. (2022) · Sensors · doi:10.3390/s22145346 · Cited by 26
“Anders et al. [23] recently argued that global XAI methods (such as TCAV) are inappropriate to discover unknown biases.”
MethodInfluentialMcKenzie, J. E., Taljaard, M., Hemming, K., Arnup, S. J., Giraudeau, B., Eldridge, S. M., Hooper, R., Kahan, B. C., Li, T., Moher, D., et al. (2025) · BMJ · doi:10.1136/bmj-2024-080472 · Cited by 36
Hellhammer, D. H., Meinlschmidt, G., & Pruessner, J. C. (2018) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2017.12.009 · Cited by 29
Variation in diagnostic NAFLD/NASH read-outs in paired liver samples from rodent models
Jensen, V. S., Tveden‐Nyborg, P., Zacho-Rasmussen, C., Quaade, M. L., Ipsen, D. H., Hvid, H., Fledelius, C., Wulff, E. M., & Lykkesfeldt, J. (2019) · Journal of Pharmacological and Toxicological Methods · doi:10.1016/j.vascn.2019.106651 · Cited by 28
Hassoun‐Kheir, N., Stabholz, Y., Kreft, J., de la Cruz, R., Dechesne, A., Smets, B. F., Romalde, J. L., Lema, A., Balboa, S., García-Riestra, C., et al. (2021) · One Health · doi:10.1016/j.onehlt.2021.100339 · Cited by 28
Kołakowski, P., Ampah, J. D., Wróbel, K., Yusuf, A. A., Gil, M., Afrane, S., Jin, C., & Liu, H. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.130651 · Cited by 26
Reproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 43
“[ 64] analyzed data from Scopus, which showed that the reproducibility problem was prevalent in several other fields as well.”
MethodArroyo-Araujo, M., Voelkl, B., Laloux, C., Novak, J., Koopmans, B., Waldron, A., Seiffert, I., Stirling, H., Aulehner, K., Janhunen, S. K., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001886 · Cited by 33
“Defining results replicability as the ability to duplicate results from a previous scientific claim supported by new data [1,2].”
ResultKoçak, B., Yardımcı, A. H., Yüzkan, S., Keles, A., Altun, O., Bulut, E., Bayrak, O. N., & Okumus, A. A. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.11.030 · Cited by 32
Cobey, K. D., Fehlmann, C. A., Franco, M. C., Ayala, A. P., Sikora, L., Rice, D. B., Xu, C., Ioannidis, J. P. A., Lalu, M. M., Ménard, A., et al. (2023) · eLife · doi:10.7554/elife.78518 · Cited by 29
Self-citationMuddled meanings hamper efforts to fix reproducibility crisis
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.20076 · Cited by 25
Getting personal with the “reproducibility crisis”: interviews in the animal research community
Fitzpatrick, B. G., Koustova, E., & Wang, Y. (2018) · Lab Animal · doi:10.1038/s41684-018-0088-6 · Cited by 31
Quantify and control reproducibility in high-throughput experiments
Zhao, Y., Sampson, M. G., & Wen, X. (2020) · Nature Methods · doi:10.1038/s41592-020-00978-4 · Cited by 29
Selective outcome reporting in root coverage randomized clinical trials
Sendyk, D. I., Vilela, N., Neto, J. B. C., Tatakis, D. N., & Pannuti, C. M. (2021) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13451 · Cited by 24
Kinds of Replicability: Different Terms and Different Functions
Matarese, V. (2022) · Axiomathes · doi:10.1007/s10516-021-09610-2 · Cited by 35
Chou, A., Espin, A. T., Kyritsis, N., Huie, J. R., Khatry, S., Funk, J., Hay, J., Lofgreen, A., Shah, R. R., McCann, C., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0265254 · Cited by 33
“However, clinical utility requires ML models to be interpretable for biological mechanisms, verifiable by clinicians, and methodologically and inferentially reproducible [15, 16].”
BackgroundReproducibility and Practical Adoption of GEOBIA with Open-Source Software in Docker Containers
Knoth, C., & Nüst, D. (2017) · Remote Sensing · doi:10.3390/rs9030290 · Cited by 29
“The Vienna Principles’ definition focuses on traceability [7]; others treat “reproducibility” and “replicability” as interchangeable [9] or completely different terms [10]; others qualify the term further, e.”
BackgroundLos desafíos que enfrentan las prácticas de ciencia abierta
Fressoli, M., & Arza, V. (2018) · Teknokultura Revista de Cultura Digital y Movimientos Sociales · doi:10.5209/tekn.60616 · Cited by 43
Replication and generalization in applied neuroimaging
Lerma‐Usabiaga, G., Mukherjee, P., Ren, Z., Perry, L. M., & Wandell, B. A. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116048 · Cited by 26
“In addition, but less appreciated, is that neuroimaging applications deployed in the field will use a range of instruments, participant populations, and measurement protocols (Goodman et al., 2016); the range of conditions in the field will be wider than that encountered in scientific studies.”
Ghezzi, P., & Mooradian, A. D. (2020) · Handbook of experimental pharmacology · doi:10.1007/164_2020_379 · Cited by 25
Perspectives on Data Reproducibility and Replicability in Paleoclimate and Climate Science
Bush, R. T., Dutton, A. L., Evans, M. N., Loft, R., & Schmidt, G. A. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.00cd8f85 · Cited by 30
“Reproducibility and replicability are considered core tenets of the scientific process and central to the creation and testing of scientific theories and models of the way the world works (Goodman et al., 2016; Stodden et al., 2016; National Academies, 2019; Powers and Hampton, 2019).”
Reproducible untargeted metabolomics workflow for exhaustive MS2 data acquisition of MS1 features
Yu, M., Dolios, G., & Petrick, L. M. (2022) · Journal of Cheminformatics · doi:10.1186/s13321-022-00586-8 · Cited by 28
Grund, S., Lüdtke, O., & Robitzsch, A. (2024) · Psychological Methods · doi:10.1037/met0000526 · Cited by 30
Brain signal predictions from multi-scale networks using a linearized framework
Hagen, E., Magnússon, S. H., Ness, T. V., Halnes, G., Babu, P. N., Linssen, C., Morrison, A., & Einevoll, G. T. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010353 · Cited by 26
Balancing sufficiency and impact in reporting standards for mass spectrometry imaging experiments
Gustafsson, O., Winderbaum, L., Condina, M. R., Boughton, B. A., Hamilton, B. R., Undheim, E. A. B., Becker, M. S., & Hoffmann, P. (2018) · GigaScience · doi:10.1093/gigascience/giy102 · Cited by 24
“apply methods or conclusions in subsequent studies, improve on workflows, and make comparisons with existing studies [11, 61].”
BackgroundMethodInfluentialOverview authors rarely defined systematic reviews that are included in their overviews
Faggion, C. M., & Diaz, K. T. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.01.004 · Cited by 24
Changing the logic of replication: A case from infant studies
Margoni, F., & Shepperd, M. (2020) · Infant Behavior and Development · doi:10.1016/j.infbeh.2020.101483 · Cited by 22
Bridgeford, E., Wang, S., Yang, Z., Wang, Z., Xu, T., Craddock, C., Dey, J. K., Kiar, G., Gray-Roncal, W., Colantuoni, C., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/802629 · Cited by 22
“The concept of reproducability is closely related to the statistical concepts of stability [2] and robustness [3].”
BackgroundMoeller, J., Dietrich, J., Neubauer, A. B., Brose, A., Kühnel, J., Dehne, M., Jaehne, M. F., Schmiedek, F., Bellhäuser, H., Malmberg, L., et al. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/5wsna · Cited by 30
A Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 34
Reproducible Research Using Biomodels
Mendes, P. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0498-z · Cited by 20
Responses of Cortical Neurons to Intracortical Microstimulation in Awake Primates
Yun, R., Mishler, J. H., Perlmutter, S. I., Rao, R. P. N., & Fetz, E. E. (2023) · eNeuro · doi:10.1523/eneuro.0336-22.2023 · Cited by 29
Catalá-López, F., Ridao-López, M., Tejedor-Romero, L., Caulley, L., Hutton, B., Husereau, D., Alonso‐Arroyo, A., Bernal‐Delgado, E., Drummond, M., & Moher, D. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.10.024 · Cited by 21
Reporting on data sharing: executive position of the EQUATOR Network
Moher, D., Collins, G. S., Hoffmann, T., Glasziou, P., Ravaud, P., & Bian, Z. (2024) · BMJ · doi:10.1136/bmj-2024-079694 · Cited by 34
“20 Transparently reported research is fundamental to examining reproducibility, 21 as is sharing data.”
Khan, M. S., Khan, M. S., Ansari, Z. N., Siddiqi, T. J., Khan, S. U., Riaz, I. B., Asad, Z. U. A., Mandrola, J. M., Wason, J. M. S., Warraich, H. J., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.3082 · Cited by 28
Gheran, B.-F., Villarreal-Narvaez, S., Vatavu, R.-D., & Vanderdonckt, J. M. (2022) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3531073.3531112 · Cited by 25
“, both the reproducibility of the results and the inferential reproducibility [19] are threatened.”
BackgroundMethodInfluentialReproducibility of prediction models in health services research
Belbasis, L., & Panagiotou, O. A. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06082-4 · Cited by 28
“Reproducibility is the ability of independent researchers to obtain the same (or similar) results when repeating an experiment or test, and it is considered a hallmark of high-quality science [8, 9].”
BackgroundGundelach, H., & Nielsen, B. B. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114252 · Cited by 23
A Laboratory Critical Incident and Error Reporting System for Experimental Biomedicine
Dirnagl, U., Przesdzing, I., Kurreck, C., & Major, S. (2016) · PLoS Biology · doi:10.1371/journal.pbio.2000705 · Cited by 23
A data-driven approach for evaluating multi-modal therapy in traumatic brain injury
Haefeli, J., Ferguson, A. R., Bingham, D., Orr, A., Won, S. J., Lam, T. I., Shi, J., Hawley, S., Liu, J., Swanson, R. A., et al. (2017) · Scientific Reports · doi:10.1038/srep42474 · Cited by 25
Spatial and temporal replicability of meteoric and in situ10Be concentrations in fluvial sediment
Gonzalez, V. S., Schmidt, A. H., Bierman, P. R., & Rood, D. H. (2017) · Earth Surface Processes and Landforms · doi:10.1002/esp.4205 · Cited by 22
Schnell, S. D. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0500-9 · Cited by 28
“They are also different between social, behavioral and economic sciences (Bollen et al 2015), computational neuroscience (Crook 2013, Plesser 2018) and biomedical sciences (Goodman et al. 2016).”
BackgroundMethodTietbohl, L. A. C., Neto, C. B. D. M., Silva, L. C. R., Dolabela, I. B., Franco, T. C., Enriquez, J. J. S., Santos, M. G., Fernandes, C. P., Machado, F. P., Mexas, R., et al. (2019) · Journal of Essential Oil Research · doi:10.1080/10412905.2019.1631894 · Cited by 28
Sensor Reproducibility Analysis: Challenges and Potential Solutions
Sekhar, P. K., Billey, W., Begay, M., Thomas, B., Woody, C., & Soundappan, T. (2024) · ECS Sensors Plus · doi:10.1149/2754-2726/ad9936 · Cited by 23
The New Reality of Reproducibility: The Role of Data Work in Scientific Research
Feinberg, M., Sutherland, W., Nelson, S. B., Jarrahi, M. H., & Rajasekar, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392840 · Cited by 28
“The notion of reproducibility has long been integral to an understanding of good scientific practice [22].”
BackgroundDeciphering the “Art” in Modeling and Simulation of the Knee Joint: Variations in Model Development
Rooks, N. B., Schneider, M., Erdemir, A., Halloran, J. P., Laz, P. J., Shelburne, K. B., Hume, D. R., Imhauser, C. W., Zaylor, W., Elmasry, S. S., et al. (2021) · Journal of Biomechanical Engineering · doi:10.1115/1.4050028 · Cited by 21
King, D. R., Hardin, K. M., Hoeker, G. S., & Poelzing, S. (2022) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00164.2022 · Cited by 22
Online Multiple Hypothesis Testing
Robertson, D. S., Wason, J. M. S., & Ramdas, A. (2023) · Statistical Science · doi:10.1214/23-sts901 · Cited by 39
“Goodman, Fanelli and Ioannidis (2016) state that “Multiplicity, combined with incomplete reporting, might be the single largest contributor to the phenomenon of nonreproducibility, or falsity, of published claims” and go on to say that “Scientific fields that routinely work with multiple hypotheses…”
BackgroundCaveat emptor: the combined effects of multiplicity and selective reporting
Li, T., Mayo-Wilson, E., Fusco, N., Hong, H., & Dickersin, K. (2018) · Trials · doi:10.1186/s13063-018-2888-9 · Cited by 21
Correcting Corrupt Research: Recommendations for the Profession to Stop Misuse of p -Values
Kmetz, J. L. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518271 · Cited by 21
“Part of the success of replication work will depend on regular monitoring and commentary on studies published on these media, especially since exact replications or reproductions of those studies may not be possible (Goodman, Fanelli, and Ioannidis 2016).”
BackgroundPsychiatric neuroimaging designs for individualised, cohort, and population studies
Gell, M., Noble, S., Laumann, T. O., Nelson, S. M., & Tervo‐Clemmens, B. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01918-y · Cited by 27
Transparency, bias, and reproducibility across science: a meta-research view
Ioannidis, J. P. A. (2024) · Journal of Clinical Investigation · doi:10.1172/jci181923 · Cited by 26
“We all seek reproducibility, but what exactly is reproducibility (22)?”
Self-citationMehrnia, S. S., Safahi, Z., Mousavi, A., Panahandeh, F., Farmani, A., Yuan, R., Rahmim, A., & Salmanpour, M. R. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01458-x · Cited by 25
Enhancing reproducibility using interprofessional team best practices
Rolland, B., Burnside, E. S., Voils, C. I., Shah, M. N., & Brasier, A. R. (2020) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.512 · Cited by 24
Mayer, A. R., Ling, J. M., Dodd, A. B., Rannou-Latella, J. G., Stephenson, D. D., Dodd, R. J., Mehos, C. J., Patton, D. A., Cullen, D. K., Johnson, V. E., et al. (2021) · Frontiers in Neurology · doi:10.3389/fneur.2021.658461 · Cited by 21
“To date, no studies have evaluated the reproducibility of head kinematics, which, by definition (65, 66), requires the exact same initial injury conditions to be repeated across multiple animals (i.”
BackgroundMethodInfluentialCollaborative methods to enhance reproducibility and accelerate discovery
Leins, D. A., Haase, S. B., Eslami, M., Schrier, J., & Freeman, J. (2023) · Digital Discovery · doi:10.1039/d2dd00061j · Cited by 25
Naaman, K., Grant, S., Kianersi, S., Supplee, L. H., Henschel, B., & Mayo-Wilson, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.221093 · Cited by 30
Post-normal institutional identities: Quality assurance, reflexivity and ethos of care
Pereira, Â. G., & Saltelli, A. (2017) · Futures · doi:10.1016/j.futures.2016.11.009 · Cited by 27
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.