What does research reproducibility mean?
The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 801–900 of 1,003
Expanding the Scope of Reproducibility Research Through Data Analysis Replications
Hofman, J. M., Goldstein, D. G., Sen, S., & Poursabzi-Sandegh, F. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366424.3383417 · Cited by 7
“…is not to debate the semantics of different terms used to categorize replication attempts, among which there is a good deal of confusion and disagreement (Christensen, Freese, & Miguel, 2019; Goodman, Fanelli, & Ioannidis, 2016; Nosek & Errington, 2019; Patil, Peng, & Leek, 2019; Plesser, 2018).”
Goodman, M. G., Li, J., Flanders, W. D., Mahood, D., Anthony, L. G., Zhang, Q., & LaKind, J. S. (2020) · Global Epidemiology · doi:10.1016/j.gloepi.2020.100040 · Cited by 4
The epistemic consequences of pragmatic value-laden scientific inference
Kubiak, A. P., & Kawalec, P. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00370-9 · Cited by 5
“…replication in the broadest sense (similar to Machery 2020) and result replication in the sense of drawing the same qualitative conclusion (Goodman et al., 2016).24 Subsequently, we propose to understand replicability from a broader sense as having two complementary and interrelated…”
BackgroundReplicability of studies following a dual‐criterion design
Rosenkranz, G. K. (2021) · Statistics in Medicine · doi:10.1002/sim.9014 · Cited by 4
“(X) and an independent standard normal variable T̃ from reverting Equation (1): 𝜃(Y) = ?”
BackgroundInfluentialAn evaluation of reproducibility and transparency indicators in orthopedic literature
Fladie, I. A., Evans, S., Checketts, J. X., Tritz, D. J., Norris, B., & Vassar, M. (2021) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12457 · Cited by 3
Proceedings of the Paris Open Science European Conference: OSEC 2022
Conference, O. S. E. (2022) · doi:10.4000/books.oep.15829 · Cited by 5
Experimental Planning and Execution
Mullane, K., Curtis, M. J., & Williams, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00002-1 · Cited by 3
Authorship Transparency in an Era of Accountability
Kerr, R. L., de Vreese, C. H., Logan, R. A., Steiner, L., Pearson, G. S., & Pierson, C. (2018) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699018802805 · Cited by 3
“Overall, comprehensive contributor disclosure (and role identification) is perceived to help address an array of contemporary challenges in medical research (Academy of Medical Sciences, 2016; Begley & Ioannidis, 2015; Goodman et al., 2016; Kenall et al., 2015; Shen & Barabasi, 2014).”
BackgroundWelty, L. J., Rasmussen, L. V., Baldridge, A. S., & Whitley, E. W. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa043 · Cited by 3
NPARS—A Novel Approach to Address Accuracy and Reproducibility in Genomic Data Science
Ma, L., Peterson, E. A., Shin, I. J., Muesse, J. L., Marino, K., Steliga, M. A., & Johann, D. J. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.725095 · Cited by 5
“Nonetheless, in several fi elds, non-reproducibility is still an obstacle towards the better understanding of datasets, further blocking the path to new scienti fi c discoveries (Mobley et al., 2013; Iqbal et al., 2016; Goodman et al., 2018; Wen et al., 2018).”
BackgroundExamining MEDLINE search query reproducibility and resulting variation in search results
Burns, C. S., Shapiro, R. M., Nix, T., & Huber, J. T. (2019) · iConference · doi:10.21900/iconf.2019.103369 · Cited by 3
“Therefore the purpose of this project is to focus on reproducible queries across these systems [6][7], and to determine 1) whether it is possible to reproduce search queries created in PubMed/MEDLINE in other MEDLINE databases; 2) to understand how and why queries may or may not be reproduced; and 3) to understand obs…”
BackgroundReplication data collection highlights value in diversity of replication attempts
DeSoto, K. A., & Schweinsberg, M. (2017) · Scientific Data · doi:10.1038/sdata.2017.28 · Cited by 3
Video can make science more open, transparent, robust, and reproduciblePreprint
Gilmore, R. O., & Adolph, K. E. (2017) · doi:10.31234/osf.io/tcfqf · Cited by 3
Guest Editorial: Background and Rationale for Clinical Trial Registration
Abrams, H. B., Chisolm, T. H., Sanchez, V. A., Arnold, M. L., & Galster, J. A. (2018) · Ear and Hearing · doi:10.1097/aud.0000000000000550 · Cited by 3
Towards a Culture of Open Science and Data Sharing in Health and Medical Research
Rowhani-Farid, A. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.119697 · Cited by 7
Sun, N., Mormino, E. C., Chen, J., Sabuncu, M. R., Yeo, B. T. T., & Initiative, A. D. N. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/462143 · Cited by 3
Computational and informatics advances for reproducible data analysis in neuroimagingPreprint
Poldrack, R. A., Gorgolewski, K. J., & Varoquaux, G. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10024 · Cited by 3
Open-source 3D printed air-jet for generating monodispersed alginate microhydrogelsPreprint
Hadley, D. J., Campbell, K. T., Gabriel, M. H., & Silva, E. A. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/804849 · Cited by 3
Braun, A. R., Liao, E. E., Horvath, M., Young, M. C., Lo, C. H., Brown, R. Z., Evans, M. D., Luk, K. C., Thomas, D. D., & Sachs, J. N. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.01.09.900845 · Cited by 3
Comment on Glenn Shafer's 'Testing by Betting'Preprint
Vovk, V. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3684664 · Cited by 5
“A serious objection to the method is that different people analyzing the same data may get very different answers (thus violating “inferential reproducibility” [6, 8]).”
MethodUsui, T., Macleod, M., McCann, S., Senior, A., & Nakagawa, S. (2020) · bioRxiv · doi:10.1101/2020.10.26.354274 · Cited by 3
Replication Redux: The Reproducibility Crisis and the Case of Deworming
Ozier, O. W. (2019) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-8835 · Cited by 4
Astrocyte-secreted glypican-4 drives APOE4-dependent tau pathologyPreprint
Saroja, S. R., Gorbachev, K., Julia, T. C. W., Goate, A., & Pereira, A. C. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.07.07.451493 · Cited by 3
Reproducibility and replicability in science: A Sisyphean task
Borges, R. M. (2022) · Journal of Biosciences · doi:10.1007/s12038-022-00259-6 · Cited by 9
“This narrow view of reproducibility apparently springs from the work of the seismologist and computer scientist Jon Claerbout (Goodman et al. 2016).”
BackgroundGiofrè, D., Boedker, I., Cumming, G., Rivella, C., & Tressoldi, P. E. (2022) · doi:10.31222/osf.io/8ya3m · Cited by 3
What does Clinical and Experimental Allergy mean by ‘Trusted Evidence in Allergy’?
Boyle, R. J., & Shamji, M. H. (2023) · Clinical & Experimental Allergy · doi:10.1111/cea.14274 · Cited by 3
Bollmann, S., Renton, A., Dao, T. T., Johnstone, T., Civier, O., Sullivan, R., White, D., Lyons, P., Slade, B., Abbott, D. F., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2649734/v1 · Cited by 3
Cobey, K. D., Ebrahimzadeh, S., Page, M. J., Thibault, R. T., Nguyen, P., Abu-Dalfa, F., & Moher, D. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.18.558287 · Cited by 3
The Role Of Software Engineering In Bioinformatics
Lawlor, B. (2021) · doi:10.34719/rjbz6161 · Cited by 2
Lemaitre, J. C., Popelka, D., Ribotta, B., Westlake, H. E., Chakrabarti, S., Xiaoxue, L., Hanson, M. A., Jiang, H., Cara, F. D., Kurant, E., et al. (2026) · eLife · doi:10.7554/elife.108403.1 · Cited by 7
Westlake, H. E., David, F., Tian, Y., Krakovic, K., Dolgikh, A., Juravlev, L., de Bournonville, T. E., Carboni, A., Melcarne, C., Shan, T., et al. (2026) · eLife · doi:10.7554/elife.108404.1 · Cited by 9
Measuring the replicability of our own research
Brown, R. E. (2024) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2024.110111 · Cited by 5
“…al., 2021 ; Arroyo-Araujo et al., 2022 ; Wilson et al., 2023) and the expected equal and opposite reaction, arguing that failure to replicate is necessary in the biological sciences (Pashler and Harris, 2012 ; Goodman et al., 2016 ; Redish et al., 2018 ; Nosek and Errington, 2020 ; Bernard, 2023 ).”
Temporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 5
Unraveling sperm kinematic heterogeneity with machine learning
Martínez, A. A. (2025) · Asian Journal of Andrology · doi:10.4103/aja202544 · Cited by 3
“The global trend toward reproducible science, 38 along with the open-source and open-hardware movements, has enabled researchers to overcome the high cost associated with proprietary scientific equipment.”
Cabrera, M. D., & del Valle, A. C. (2025) · Frontiers in Lab on a Chip Technologies · doi:10.3389/frlct.2025.1669220 · Cited by 3
Replicable Bandits for Digital Health Interventions
Zhang, K. W., Closser, N., Trella, A. L., & Murphy, S. A. (2025) · Statistical Science · doi:10.1214/25-sts1017 · Cited by 18
“This is replicability of the statistical inferential methods across trial replications [26].”
Ten common mistakes that could ruin your enrichment analysis
Bora, A., McKenzie, M., & Ziemann, M. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014122 · Cited by 4
“According to Goodman and colleagues [36], methods reproducibility is: “the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated.””
Brodeur, A., Valenta, D., Marcoci, A., Aparicio, J. P., Mikola, D., Barbarioli, B., Alexander, R., Deer, L., Stafford, T., Vilhuber, L., et al. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2524747123 · Cited by 16
Advancing reproducibility and replicability in simulation: Challenges and opportunities
Huang, Y., & Çetinkaya, D. (2026) · SIMULATION · doi:10.1177/00375497261444678 · Cited by 2
“2,4,30 It calls for a way of working and thinking about how we develop and maintain our scientific simulations.”
InfluentialKaptein, M. C., van Emden, R., & Iannuzzi, D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0174182 · Cited by 3
Nanothera(g)nosis and Chemistry: A Fruitful Binomial
Valcárcel, M. (2018) · Journal of Nanomedicine & Nanotechnology · doi:10.4172/2157-7439.1000490 · Cited by 2
Transparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 3
Scholarly Publishing and Scientific Reproducibility
Bomzon, A., & Tobin, G. (2021) · Laboratory animal science and medicine · doi:10.1007/978-3-030-66147-2_9 · Cited by 3
Programas de intervención y replicabilidad: consideraciones sobre su evaluación en psicología
Romo, J. F. M. (2021) · Revista Iberoamericana de Psicología · doi:10.33881/2027-1786.hrip.14108 · Cited by 6
Descriptive Epidemiology of Hospitalization of Patients with a Rare Tumor in an Italian Region
Rosa, A., Fontana, V., Filiberti, R. A., Pronzato, P., & Mannucci, M. (2022) · Current Oncology · doi:10.3390/curroncol29120762 · Cited by 2
“Although this confirms the originality of our analysis, it definitely impedes any, albeit informal, meta-comparisons [31].”
ResultA Study in Reproducibility: The Congruent Matching Cells Algorithm and cmcR Package
Zemmels, J., VanderPlas, S., & Hofmann, H. (2023) · The R Journal · doi:10.32614/rj-2023-014 · Cited by 3
“Reproducibility is an indispensable component of scientific validity (Goodman, Fanelli, and Ioannidis 2016).”
BackgroundVorteile und Ansätze eines Qualitätsmanagementsystems in biomedizinischen Forschungslaboratorien
Brünschwitz, S., & Kleymann-Hilmes, J. (2023) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-023-03797-y · Cited by 3
“Bemerkenswert ist, dass die Begrif-fe „Reproduzierbarkeit“ und „Replizier-barkeit“ keine einheitliche Definition in-nehaben ([15]”
BackgroundNoninvasive total counting of cultured cells using a home-use scanner with a pattern sheet
Mizuno, M., Maeda, Y., Sanami, S., Matsuzaki, T., Yoshikawa, H., Ozeki, N., Koga, H., & Sekiya, I. (2024) · iScience · doi:10.1016/j.isci.2024.109170 · Cited by 5
Using Artificial Populations to Study Psychological Phenomena in Neural Models
Roberts, J., Moore, K., Wilenzick, D., & Fisher, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29856 · Cited by 13
“Without appropriate experimental methodology conclusions may not be robust in the face of variations, a symptom associated with the greater replicability crisis (Goodman, Fanelli, and Ioannidis 2016).”
BackgroundKou, T. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658951 · Cited by 7
“In the context of sciences more broadly, [42]’s notion of inferential replicability interprets results as qualitative conclusions , [47]’s notion of interpretation replicability .”
Application of preclinical juvenile animal models
Hermans, E., Toelen, J., Ventrella, D., Schaepdrijver, L. D., Turner, K. J., Croubels, S., & Devreese, M. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-88459-4.00009-2 · Cited by 3
Braczko, F., Skyschally, A., Lieder, H. R., Kather, J. N., Kleinbongard, P., & Heusch, G. (2024) · Basic Research in Cardiology · doi:10.1007/s00395-024-01081-x · Cited by 4
Documenting research in simulation science to enhance understanding for reusability
Hermann, S., & Fehr, J. (2024) · Royal Society Open Science · doi:10.1098/rsos.240776 · Cited by 2
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 6
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
Public engagement with research reproducibility
Vinatier, C., Kozula, M., den Eynden, V. V., Caquelin, L., Roubík, H., Stegeman, I., & Naudet, F. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002953 · Cited by 2
Bianco, R., Ferraroni, M., Speciani, M. C., Parpinel, M. T., & Edefonti, V. C. (2025) · Advances in Nutrition · doi:10.1016/j.advnut.2025.100397 · Cited by 2
“Focused on promoting reproducible research [9], our group has contributed to clarifying terminology around the reproducibility, validity, and reliability of a posteriori DPs [10,11] — those derived from multivariate statistical methods such as principal component analysis (PCA), exploratory factor…”
Lindsay, R. M. (2025) · Advances in management accounting · doi:10.1108/s1474-787120250000037001 · Cited by 9
Systematic refinement of experimental practices to improve repeatability in flow battery cycling
O’Connor, H., Quinn, A. H., Brushett, F. R., Istrate, O. M., Glover, S., Bailey, J. J., & Nockemann, P. (2025) · Discover Electrochemistry. · doi:10.1007/s44373-025-00036-8 · Cited by 6
Journal Data Accessibility Policies: Challenges and Opportunities
Тихонова, Е. В., & Косычева, М. А. (2024) · Health Food & Biotechnology · doi:10.36107/hfb.2024.i4.s251 · Cited by 3
Open Science in Three Acts: Foundations, Practice, and Implementation - Third Act
Rogers, P., & Limongi, R. (2025) · BAR - Brazilian Administration Review · doi:10.1590/1807-7692bar2025250162 · Cited by 2
Learning from Principles of Evidence-Based Medicine to Optimize Nonclinical Research Practices
Lefevre, I. A., & Balice‐Gordon, R. J. (2019) · Handbook of experimental pharmacology · doi:10.1007/164_2019_276 · Cited by 4
Popova, K. (2021) · Science Technology & Human Values · doi:10.1177/01622439211032392 · Cited by 3
Gerring, J., Karcher, S., & Apfeld, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.015 · Cited by 3
Haring, R., & Bell, R. J. (2018) · Climacteric · doi:10.1080/13697137.2018.1476968 · Cited by 3
Designing Structured Design Thinking Framework for Societal System Design in the Unknown Context
Tomita, Y., & Maéno, T. (2018) · Journal of Management and Strategy · doi:10.5430/jms.v9n3p103 · Cited by 2
Why the p-value is under fire?
Sapra, R. L., & Nundy, S. (2018) · Current Medicine Research and Practice · doi:10.1016/j.cmrp.2018.10.003 · Cited by 4
Conducting Replication Studies With Confidence
Morin, K. H. (2018) · Journal of Nursing Education · doi:10.3928/01484834-20181022-02 · Cited by 2
Reproducible Findings in Systematic Reviews and Meta-analyses in Oncology
Unger, J. M. (2019) · JAMA Oncology · doi:10.1001/jamaoncol.2019.2664 · Cited by 2
What’s Wrong with Replicating the Old Boys’ Networks?
Teele, D. L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.016 · Cited by 2
Giuliani, G., Câmara, G., Killough, B., & Minchin, S. (2020) · MDPI eBooks · doi:10.3390/books978-3-03928-093-3 · Cited by 3
Wainer, J. M. G., Fardella, C., & Espinosa‐Cristia, J. F. (2021) · History & Philosophy of the Life Sciences · doi:10.1007/s40656-021-00418-2 · Cited by 8
The imperative to find the courage to redesign the biomedical research enterprise
Bielekova, B., & Brownlee, S. (2021) · F1000Research · doi:10.12688/f1000research.53654.1 · Cited by 3
Effect of phosphorus and potassium application on growth, yield and quality of kinnow mandarin.
Dudi, O. P., Singh, R., Singh, S., & Baloda, S. (2007) · Haryana journal of horticultural sciences · Cited by 3
Rigour and reproducibility in Canadian research: call for a coordinated approach
Thibault, R. T., Munafo, M. R., & Moher, D. (2022) · FACETS · doi:10.1139/facets-2021-0162 · Cited by 4
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.013 · Cited by 2
Lieberman, E. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.003 · Cited by 3
Pro-Con Debate: Should Code Sharing Be Mandatory for Publication?
Melvin, R. L., Barker, S. J., Kiani, J., & Berkowitz, D. E. (2022) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000005848 · Cited by 2
A taxonomy of tools and approaches for distributed genomic analyses
Garzón, W., Benavides, L. G. B., Gaignard, A., Redon, R., & Südholt, M. (2022) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2022.101024 · Cited by 2
Gayle, V., & Connelly, R. (2022) · Methodological Innovations · doi:10.1177/20597991221111681 · Cited by 2
“This is not to suggest that new publication guidelines are a panacea, indeed some evaluations for example Herndon and O’Reilly (2016) suggest that while journals are adopting new policies they are doing so in an incomplete and varied manner.”
BackgroundMethodVerykouki, E., & Nakas, C. T. (2023) · Stats · doi:10.3390/stats6020035 · Cited by 4
“Replicability refers to the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected [53,54].”
BackgroundReproducibility of Indian DH Projects: A case study
Justin, J., & Menon, N. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00071-0 · Cited by 3
“That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results” (Cited in Goodman et al., 2016).”
InfluentialResearch rigor and reproducibility in research education: A CTSA institutional survey
Axfors, C., Malički, M., & Goodman, S. N. (2024) · Journal of Clinical and Translational Science · doi:10.1017/cts.2024.10 · Cited by 2
“Computational reproducibility refers to the process of obtaining the same (statistical) results by re-running the published analysis using the researchers ’ methods and (deposited) code or data [9].”
MethodSelf-citationReproducibility in medical image computing: what is it and how is it assessed?
Colliot, O., Thibeau–Sutre, E., Brianceau, C., & Burgos, N. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-323761-4.00018-3 · Cited by 5
On the Implementation of Centerline Extraction Based on Confidence Vote in Accumulation Map
Kerautret, B., Krähenbühl, A., Debled-Rennesson, I., & Lachaud, J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56414-2_9 · Cited by 2
Letcher, B. H., Walker, J. D., O’Donnell, M. J., Whiteley, A. R., Nislow, K. H., & Coombs, J. A. (2018) · Fisheries · doi:10.1002/fsh.10067 · Cited by 3
Reproducibility of young learners' susceptibility to the learning context
Merkelbach, I., Plak, R. D., & Rippe, R. C. A. (2018) · Learning and Individual Differences · doi:10.1016/j.lindif.2018.05.020 · Cited by 3
Les études portant sur les bases de données médico-administratives sont-elles reproductibles ?
Looten, V. (2019) · médecine/sciences · doi:10.1051/medsci/2019134 · Cited by 2
Creating optimal conditions for reproducible data analysis in R with ‘fertile’
Bertin, A. M., & Baumer, B. S. (2021) · Stat · doi:10.1002/sta4.332 · Cited by 2
Alternative Facts? Antidepressants and Falls in Older Adults
Lenze, E. J., Stark, S. L., & Avidan, M. S. (2017) · American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2017.07.017 · Cited by 2
Quality and denoising in real-time fMRI neurofeedback: a methods review
Heunis, S. (2019) · TU/e Research Portal · doi:10.17605/osf.io/2n5p3 · Cited by 4
From one environment to many: The problem of replicability of statistical inferencesPreprint
Higgins, J. J., Higgins, M., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.10036 · Cited by 2
“reproducibility crisis [Pashler and Wagenmakers, 2012] has led to the use of many nearly{synonymous notions of replicability including: reproducibility, reliability, robustness, and generalizability [Goodman et al., 2016]. In this paper, we are most concerned with the idea of replicability dened as in Bollen et al. [2…”
ResultReproducibility Revisited: Reflections of an Editor
Miller, G. W. (2019) · Toxicological Sciences · doi:10.1093/toxsci/kfz118 · Cited by 2
Zeevi, Y., Astashenko, S., Mudrik, L., & Benjamini, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11585 · Cited by 4
Khan, K., Luo, H., & Xi, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.01285 · Cited by 3
“There are various definitions of reproducibility and replicability in use, varying depending on the field 30;31;32”
Open science practices and risks arising from misuse of biological researchPreprint
Smith, J. A., & Sandbrink, J. B. (2021) · doi:10.31222/osf.io/8afwn · Cited by 2
Prabhu, A., & Fox, P. (2021) · Encyclopedia of earth sciences series/Encyclopedia of earth sciences · doi:10.1007/978-3-030-26050-7_277-1 · Cited by 3
Organizing laboratory information to analyze the reproducibility of experimental workflowsPreprint
Peccoud, J., Johnson, D., Peccoud, S., Setchell, J., & Zhou, W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.05.487214 · Cited by 2
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.