What does research reproducibility mean?
The DOI matches its Crossref record; 38 of the 41 references it cites were found in a registry or an index.
Abstract
The language and conceptual framework of “research reproducibility” are nonstandard and unsettled across the sciences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“High quality reporting is an important step when considering issues related to reproducibility [50].”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [50]
In the citing paper
“These factors are inherent to modern computational pipelines in deep learning (Desai et al., 2025; Goodman et al., 2016; Semmelrock et al., 2025)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Goodman et al., 2016
In the citing paper
“for reasons that have nothing to do with questionable research and publication practices (28).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
In the citing paper
“the reproducibility of research methods can in principle be expected to be 100%”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [28]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“The classic file-drawer publication bias problem (wherein nonsignificant or “uninteresting” results are not published) (29) results in bias under the assumption that multiple studies are being produced independently but a biased sample is published.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
Afgan, E., Baker, D., van den Beek, M., Blankenberg, D., Bouvier, D., Čech, M., Chilton, J., Clements, D., Coraor, N., Eberhard, C., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw343 · Cited by 2,148
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
A Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Research works printed but not linked: 2
- S. N. Goodman J. Gerson Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality Rockville MD 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
- E. E. Leamer Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley Hoboken NJ 1978).
Cited by
Items 501–600 of 1,003
The Potassium Cycle and Its Relationship to Recommendation Development
Brouder, S. M., Volenec, J. J., & Murrell, T. S. (2020) · Springer eBooks · doi:10.1007/978-3-030-59197-7_1 · Cited by 20
“While numerous factors can contribute to irreproducible or non-generalizable results, solutions for improving the quality of science consistently stress complete reporting inclusive of data access (Begley and Ioannidis 2015; Button et al. 2013; Goodman et al. 2016).”
BackgroundMEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms
Burns, C. S., Nix, T., Shapiro, R. M., & Huber, J. T. (2021) · PLoS ONE · doi:10.1371/journal.pone.0234221 · Cited by 14
Sim2Ls: FAIR simulation workflows and data
Hunt, M., Clark, S. M., Mejia, D., Desai, S., & Strachan, A. H. (2022) · PLoS ONE · doi:10.1371/journal.pone.0264492 · Cited by 20
Ruiz, B., Broadbent, J. M., Thomson, W. M., Ramrakha, S., Boden, J. M., Horwood, L. J., & Poulton, R. (2022) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.12772 · Cited by 16
“This is desirable because reproducibility remains a challenging issue in contemporary science 22 and, here, we aim to address this directly.”
BackgroundEpistemic Functions of Replicability in Experimental Sciences: Defending the Orthodox View
Sikorski, M., & Andreoletti, M. (2023) · Foundations of Science · doi:10.1007/s10699-023-09901-4 · Cited by 18
“Replicability is used in both basic senses (for the first see e.g., Schwab et al., 2000; for the second see Goodman et al., 2016).”
MethodReplicability Across Multiple Studies
Bogomolov, M., & Heller, R. (2023) · Statistical Science · doi:10.1214/23-sts892 · Cited by 29
“Various definitions of replicability, and no-replicability, were considered in Nosek et al. (2022); Hung and Fithian (2020); Mathur and VanderWeele (2019); Patil et al. (2019); Hedges and Schauer (2019b); Goodman et al. (2016); Simonsohn (2015) and references within.”
BackgroundHow can my research paper be useful for future meta-analyses on forest restoration plantations?
Andivia, E., Villar‐Salvador, P., Oliet, J. A., Puértolas, J., & Dumroese, R. K. (2018) · New Forests · doi:10.1007/s11056-018-9631-y · Cited by 15
Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents
Tani, J., Daniele, A. F., Bernasconi, G., Camus, A., Petrov, A., Courchesne, A., Mehta, B., Suri, R., Zaluska, T. J., Walter, M. R., et al. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341677 · Cited by 14
“in objective is to achieve “reproducibility”. At an abstract level, reproducibility is the ability to reduce variability in the evaluation of a system such that experimental results are more credible [19]. More specifically, there are three distinct aspects to reproducibility, all equally important: 1) An experiment is…”
BackgroundVõikar, V. (2020) · Journal for Reproducibility in Neuroscience · doi:10.31885/jrn.1.2020.1375 · Cited by 16
“Moreover, reproducibility can be discussed at the level of methods, analysis, results, or conclusions (1, 2).”
BackgroundWeißgerber, S. C., Brunmair, M., & Rummer, R. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09579-1 · Cited by 17
Statistical Inference in Behavioral Research: Traditional and Bayesian Approaches
Etz, A. J., Goodman, S. N., & Vandekerckhove, J. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0007 · Cited by 13
Self-citationReproducibility improves exponentially over 63 years of social learning researchPreprint
Minocher, R., Atmaca, S., Bavero, C., McElreath, R. L., & Beheim, B. (2020) · doi:10.31234/osf.io/4nzc7 · Cited by 13
Koçak, B., Keleş, A., & Köse, F. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.232604 · Cited by 12
Multi-Institution Research and Education Collaboration Identifies New Antimicrobial Compounds
Fuller, A. A., Dounay, A. B., Schirch, D. M., Rivera, D. G., Hansford, K. A., Elliott, A. G., Zuegg, J., Cooper, M. A., Blaskovich, M. A. T., Hitchens, J. R., et al. (2020) · ACS Chemical Biology · doi:10.1021/acschembio.0c00732 · Cited by 13
Generating Small Areal Synthetic Microdata from Public Aggregated Data Using an Optimization Method
Lin, Y., & Xiao, N. (2023) · The Professional Geographer · doi:10.1080/00330124.2023.2207640 · Cited by 12
“…qualified researchers might have access to such data and are not constrained to using public aggregated data, their research might not be methodologically reproducible because it is difficult for others to collect the same data and replicate their research (Goodman, Fanelli, and Ioannidis 2016).”
BackgroundMapping strategies towards improved external validity in preclinical translational research
Carneiro, C. F. D., Drude, N., Hülsemann, M., Collazo, A., & Toelch, U. (2023) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2023.2251886 · Cited by 17
“The reproducibility of these efficacy studies (results reproducibility, sensu [2]) and, consequently, translational success depend on the reliability, internal and external validity of preclinical experiments [3–5].”
BackgroundMeiser, M., & Zinnikus, I. (2024) · Energies · doi:10.3390/en17091992 · Cited by 22
Demidenko, M. I., Mumford, J. A., & Poldrack, R. A. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00262 · Cited by 18
Making PBPK models more reproducible in practice
Domínguez-Romero, E., Mazurenko, S., Scheringer, M., dos Santos, V. A. P. M., Evelo, C. T., Anton, M., Hancock, J. M., Županič, A., & Diez, M. S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae569 · Cited by 16
“…reproducibility’: the ability for an independent study to reach the same conclusions after either replicating or reanalyzing the original study [19, 23]. ordinary differential equations (ODE) and include a realistic representation of relevant anatomical and physiological traits in the organism.”
BackgroundGupta, A., Singh, V., Chandra, S., & Garg, R. (2025) · Journal of Ayurveda and Integrative Medicine · doi:10.1016/j.jaim.2025.101157 · Cited by 15
“Furthermore, research reproducibility was used to assess the devices and equipment [42,43].”
Reproducibility: A Primer on Semantics and Implications for Research
Pellizzari, E. D., Lohr, K. N., Blatecky, A. R., & Creel, D. V. (2017) · doi:10.3768/rtipress.2017.bk.0020.1708 · Cited by 13
Reproducibility in Machine Learning for Medical Imaging
Colliot, O., Thibeau–Sutre, E., & Burgos, N. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_21 · Cited by 25
Talbot, S. R., Kumstel, S., Schulz, B., Tang, G., Abdelrahman, A., Seume, N., Wendt, E. H. U., Eichberg, J., Häger, C., Bleich, A., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-29623-8 · Cited by 17
Nichols, W. W., Bradford, P. A., & Stone, G. G. (2023) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkad049 · Cited by 13
Open Science Standards at Journals that Inform Evidence-Based Policy
Grant, S., Mayo‐Wilson, E., Kianersi, S., Naaman, K., & Henschel, B. (2023) · Prevention Science · doi:10.1007/s11121-023-01543-z · Cited by 22
“These limitations are partly attributable to detrimental research practices associated with closed science—particularly undisclosed multiple hypothesis testing and the selective non-reporting of results (Goodman et al., 2016).”
BackgroundKlau, S., Schönbrodt, F. D., Patel, C. J., Ioannidis, J. P. A., Boulesteix, A., & Hoffmann, S. (2023) · Meta-Psychology · doi:10.15626/mp.2020.2556 · Cited by 18
Self-citationA Systems Approach to Understanding and Improving Research Integrity
Gorman, D. M., Elkins, A. D., & Lawley, M. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-017-9986-z · Cited by 14
Null hypothesis significance testing and effect sizes: can we ‘effect’ everything … or … anything?
Lovell, D. P. (2020) · Current Opinion in Pharmacology · doi:10.1016/j.coph.2019.12.001 · Cited by 20
Oliveira, N. L., Botton, C. E., Nardi, A. T. D., & Umpierre, D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01845-9 · Cited by 13
“Such evidence of lowquality practices affects reproducibility, since it makes findings less verifiable, credible, and informative [17], with likely waste of financial or non-financial resources [18].”
BackgroundEditorial: Reproducibility and Rigour in Computational Neuroscience
Crook, S. M., Davison, A. P., McDougal, R. A., & Pleßer, H. E. (2020) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2020.00023 · Cited by 12
“Tennøe et al. provide the community with Uncertainpy, which is a Python toolbox for uncertainty quantification and sensitivity analysis, and also provide examples of its use with models simulated with both NEURON (Hines et al., 2020) and NEST (Gewaltig and Diesmann, 2007).”
MethodOpen and reproducible neuroimaging: from study inception to publicationPreprint
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · doi:10.31219/osf.io/pu5vb · Cited by 12
Stefani, C. M., de Lima, A. D. A., Stefani, F. M., Kung, J. Y. C., Compton, S. M., & Flores‐Mir, C. (2025) · European Journal of Orthodontics · doi:10.1093/ejo/cjaf024 · Cited by 11
“Needless to say, reliable and replicable results are essential to establish the effectiveness of an intervention [43]—clinicians need confidence that the chosen intervention consistently delivers clinically meaningful results compared to other interventions or no treatment, ideally in a…”
Steegmans, P. A. J., Bipat, S., & Reynders, R. M. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1000-1 · Cited by 12
Bench to Bedside: Modelling Inflammatory Arthritis
Chidomere, C. I., Wahid, M., Kemble, S., Chadwick, C., Thomas, R., Hardy, R., McGettrick, H. M., & Naylor, A. J. (2022) · Discovery Immunology · doi:10.1093/discim/kyac010 · Cited by 13
“They aim to improve the standard of reporting and over time the standard of experimental design to address the reproducibility crisis in biomedical science [88–91].”
BackgroundRecommendations to improve use and reporting of statistics in animal experiments
Rowe, A. (2022) · Laboratory Animals · doi:10.1177/00236772221140669 · Cited by 15
“Other causes of low p-values are more insidious.(41) Multiplicity means multiple comparisons and includes testing many hypotheses in one experiment and testing one hypothesis many times or in multiple ways in one or more studies.”
BackgroundThe primary pharmacology of ceftazidime/avibactam: resistancein vitro
Nichols, W. W., Lahiri, S. D., Bradford, P. A., & Stone, G. G. (2023) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkac449 · Cited by 11
Stark, B. C., & Dalton, S. G. (2024) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.13028 · Cited by 16
“Goodman et al. (2016) proposed a new lexicon for research reproducibility, highlighting methods reproducibility as providing sufficient detail about procedures and data so that the same procedures could be exactly repeated.”
MethodWilliams, M. J., Mullane, K., & Curtis, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00005-7 · Cited by 13
Statistical significance and p-values: guidelines for use and reporting
Parsons, N., Carey-Smith, R. L., Dritsaki, M., Griffin, X. L., Metcalfe, D., Perry, D. C., Stengel, D., & Costa, M. L. (2019) · The Bone & Joint Journal · doi:10.1302/0301-620x.101b10.bjj-2019-0890 · Cited by 11
Quality Considerations When Using Tissue Samples for Biomarker Studies in Cancer Research
Speirs, V. (2021) · Biomarker Insights · doi:10.1177/11772719211009513 · Cited by 12
An open developmental science will be more rigorous, robust, and impactful
Gilmore, R. O., & Qian, Y. (2021) · Infant and Child Development · doi:10.1002/icd.2254 · Cited by 8
Using ReproZip for Reproducibility and Library Services
Steeves, V., Rampin, R., & Chirigati, F. (2017) · IASSIST Quarterly · doi:10.29173/iq18 · Cited by 12
Nazariadli, S., Morais, D. B., Supak, S., Baran, P. K., & Bunds, K. S. (2019) · Methodological Innovations · doi:10.1177/2059799119832194 · Cited by 11
Straßer, T., Wagner, S., & Zrenner, E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234330 · Cited by 11
Measuring reproducibility of virus metagenomics analyses using bootstrap samples from FASTQ-files
Saremi, B., Kohls, M., Liebig, P., Siebert, U., & Jung, K. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa926 · Cited by 11
Expanding the scope of reproducibility research through data analysis replications
Hofman, J. M., Goldstein, D. G., Sen, S., Poursabzi-Sangdeh, F., Allen, J. N. L., Dong, L. L., Fried, B., Gaur, H., Hoq, A., Mbazor, E., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.11.003 · Cited by 16
Popp, J., & Biskup, T. (2022) · Chemistry - Methods · doi:10.1002/cmtd.202100097 · Cited by 12
Cheng, Y., Kim, M., Chen, Y., & Zhao, Y. (2026) · Coordination Chemistry Reviews · doi:10.1016/j.ccr.2026.217574 · Cited by 11
Partnering With Authors to Enhance Reproducibility at JASA
Wrobel, J., Hector, E. C., Crawford, L. A., McGowan, L. D., da Silva, N., Goldsmith, J., Hicks, S. C., Kane, M. J., Lee, Y., Mayrink, V. D., et al. (2024) · Journal of the American Statistical Association · doi:10.1080/01621459.2024.2340557 · Cited by 12
Randomized clinical trials in periodontology: focus on outcomes selection
Pannuti, C. M., Costa, F. O., Vilela, N., Retamal‐Valdes, B., Costa, A. A., Susin, C., & Feres, M. (2021) · Brazilian Oral Research · doi:10.1590/1807-3107bor-2021.vol35.0100 · Cited by 11
Reproducibility efforts as a teaching tool: A pilot study
Karathanasis, N., Hwang, D., Heng, V., Abhimannyu, R., Slogoff-Sevilla, P., Buchel, G., Frisbie, V., Li, P., Kryoneriti, D., & Rigoutsos, I. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010615 · Cited by 14
“An alternative term, Methods reproducibility, refers to the same problem but has a broader meaning [10], quoting from Ioannidis’ paper, “refers to the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated.”
BackgroundMahapatra, S., Sarbahi, A., Punia, N., Joshi, A., Avni, A., Walimbe, A., & Mukhopadhyay, S. (2023) · Journal of Biological Chemistry · doi:10.1016/j.jbc.2023.104654 · Cited by 12
Cariaga-Martínez, A. E., & Alelú‐Paz, R. (2016) · Journal of Negative Results in BioMedicine · doi:10.1186/s12952-016-0064-x · Cited by 11
Barwich, A. (2018) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2018.02.005 · Cited by 11
Oludare, S. O., Pater, M. L., Rosenblatt, N. J., & Grabiner, M. D. (2018) · Gait & Posture · doi:10.1016/j.gaitpost.2018.02.001 · Cited by 10
Pereira, L. (2019) · Data · doi:10.3390/data4030127 · Cited by 10
Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics
McManus, E., Turner, D., & Sach, T. (2019) · PharmacoEconomics · doi:10.1007/s40273-019-00836-y · Cited by 13
“Most often, these distinguish between broad and narrow replications [7, 22, 23], where a broad replication can be defined as using data from other “periods, countries, regions or other entities as appropriate” to see if the empirical finding can be repeated [24].”
BackgroundHow to Conduct Responsible Research: A Guide for Graduate Students
Antes, A. L., & Maggi, L. B. (2021) · Current Protocols · doi:10.1002/cpz1.87 · Cited by 13
“Reproducibility is the ability to duplicate results (Goodman, Fanelli, & Ioannidis, 2016).”
BackgroundButler, R. C. (2021) · Annals of Applied Biology · doi:10.1111/aab.12734 · Cited by 11
Ioannidis, J. P. A. (2019) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-019-09621-y · Cited by 10
Self-citationEvaluation of reproducible and transparent research practices in pulmonology
Smith, C., Nolan, J., Tritz, D., Heavener, T. E., Pelton, J., Cook, K., & Vassar, M. B. (2021) · Pulmonology · doi:10.1016/j.pulmoe.2020.07.001 · Cited by 10
“Reproducibility----the ability to duplicate a study’s results using the same materials and methods as the original investigator----is central to the scientific method.(1) Study reproducibility establishes confidence in the efficacy of therapies, while results that contradict original findings may lead to overturning p…”
ResultReplication: Challenges in using Data Logs to Validate Phishing Detection Ability MetricsPreprint
Canfield, C., Davis, A. L., Fischhoff, B., Forget, A., Pearman, S., & Thomas, J. (2017) · OSF Preprints (OSF Preprints) · Cited by 23
The SONATA Data Format for Efficient Description of Large-Scale Network ModelsPreprint
Dai, K., Hernando, J., Billeh, Y. N., Gratiy, S. L., Planas, J., Davison, A. P., Durá-Bernal, S., Gleeson, P., Devresse, A., Dichter, B., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/625491 · Cited by 10
Wu, J. F., Nivargi, R., Lanka, S. S. T., Menon, A. M., Modukuri, S. A., Nakshatri, N. S., Wei, X., Wang, Z., Caverlee, J., Rajtmajer, S. M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04580 · Cited by 22
“Of note, in this work, we focus on feature extraction rather than the prediction model, so we do not distinguish between reproducibility and replicability [14].”
Ting, C., & Greenland, S. (2024) · Science Communication · doi:10.1177/10755470241239947 · Cited by 16
“…labels (statistically significant vs. statistically insignificant) and treat hypothesis testing as if it could turn a continuous probabilistic phenomenon into a deterministic dichotomy, or at least one with high signal-to-noise ratio in well-controlled experiments (Goodman et al., 2016).”
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 11
Sio, F. D., Hanke, F. D., Warnke, K., Marazia, C., Galligioni, V., Fiorito, G., Stravidou, I., & Ponte, G. (2020) · Frontiers in Physiology · doi:10.3389/fphys.2020.00645 · Cited by 9
Online Data Articles: The Language of Intersubjective Stance in a Rhetorical Hybrid
Pérez‐Llantada, C. (2022) · Written Communication · doi:10.1177/07410883221087486 · Cited by 13
“…processes as a response to “concerns over reproducibility” (Kwok, 2018, p. 269) that put into question the reliability of research if the results cannot be reproduced and if the data are not fully valid and therefore not sufficiently reliable, robust, or generalizable (Goodman et al., 2016).”
BackgroundThe Problem of Irreproducible Bioscience Research
Flier, J. S. (2022) · Perspectives in biology and medicine · doi:10.1353/pbm.2022.0032 · Cited by 12
A Study on Reproducibility and Replicability of Table Structure Recognition Methods
Ajayi, K., Choudhury, M. H., Rajtmajer, S. M., & Wu, J. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41679-8_1 · Cited by 14
Hankenson, F. C., Prager, E. M., & Berridge, B. R. (2024) · Annual Review of Animal Biosciences · doi:10.1146/annurev-animal-021022-043531 · Cited by 10
Spence, J. R., & Stanley, D. J. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231217932 · Cited by 14
“In contrast, reproducibility generally refers to the ability of a researcher to generate the same results of a study from the same raw data (Goodman et al., 2016; Gundersen, 2021; Patil et al., 2019; Plesser, 2018).”
BackgroundGender Bias Detection in Court Decisions: A Brazilian Case Study
Benatti, R. M., Severi, F. C., Avila, S., & Colombini, E. L. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658937 · Cited by 19
“Reproducibility is a critical quality of modern research [2, 28, 36], given its role in scientific scrutiny, fraud prevention and detection, and strengthening of research communities, which upholds the purpose of science as an endeavor of public interest.”
BackgroundNévéol, A., Cohen, K. B., Grouin, C., & Robert, A. (2016) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/w16-6110 · Cited by 11
Why published research is untrustworthy
Lose, G., & Klarskov, N. (2017) · International Urogynecology Journal · doi:10.1007/s00192-017-3389-1 · Cited by 10
“^ The poor quality of medical research is not a new criticism [2]; however, concern has been expressed within a broad field of specialties in parallel with reports that studies are fraught with problems including poor reproducibility [3].”
BackgroundBuilding capacity to encourage research reproducibility and #MakeResearchTrue
Rethlefsen, M. L., Lackey, M. J., & Zhao, S. (2018) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2018.273 · Cited by 10
“generalizability are separate components of reproducibility [7].”
BackgroundTommaso, P. D., Floden, E. W., Magis, C., Palumbo, E., & Notredame, C. (2017) · Biologie Aujourd hui · doi:10.1051/jbio/2017029 · Cited by 12
Search results outliers among MEDLINE platforms
Burns, C. S., Shapiro, R. M., Nix, T., & Huber, J. T. (2019) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2019.622 · Cited by 11
WURS: a simulation software for university rankings—software review
Sınıksaran, E., & Satman, M. H. (2019) · Scientometrics · doi:10.1007/s11192-019-03269-8 · Cited by 11
Computational experiment design for operations model simulation
Lara, J. D., Lee, J. T., Callaway, D. S., & Hodge, B. S. (2020) · Electric Power Systems Research · doi:10.1016/j.epsr.2020.106680 · Cited by 10
Seeman, J. I. (2018) · Foundations of Chemistry · doi:10.1007/s10698-018-9304-0 · Cited by 11
Knock, E., Whittles, L. K., Perez‐Guzman, P. N., Bhatia, S., Guntoro, F., Watson, O. J., Whittaker, C., Ferguson, N. M., Cori, A., Baguelin, M., et al. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16466.1 · Cited by 9
The SONATA Data Format for Efficient Description of Large-Scale Network ModelsPreprint
Dai, K., Hernando, J., Billeh, Y. N., Gratiy, S. L., Planas, J., Davison, A. P., Durá-Bernal, S., Gleeson, P., Devresse, A., Gevaert, M., et al. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3387685 · Cited by 8
Progress toward openness, transparency, and reproducibility in cognitive neurosciencePreprint
Gilmore, R. O., Diaz, M. T., Wyble, B., & Yarkoni, T. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/5veew · Cited by 11
Data sharing and transparency indicators in published RCTs in Oral Health between 2017 and 2023
Mikelis, F., Karamalaki, D., Mikeli, A., Tzanetakis, G. N., & Koletsi, D. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105263 · Cited by 8
Bearing a Bag-of-Tales: An Open Corpus of Annotated Folktales for Reproducible Research
Hagedorn, J., & Darányi, S. (2022) · Journal of Open Humanities Data · doi:10.5334/johd.78 · Cited by 16
From crisis to opportunity through innovation
Montes, J. O., Gómez-Cruz, N. A., Batz, A., Cárdenas, L. F. S., & Holguín, H. M. (2024) · Management Research Review · doi:10.1108/mrr-05-2023-0324 · Cited by 12
The characteristics and reproducibility of motor speech functional neuroimaging in healthy controls
Kenyon, K. H., Boonstra, F. M. C., Noffs, G., Morgan, A. T., Vogel, A. P., Kolbe, S., & van der Walt, A. (2024) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2024.1382102 · Cited by 9
Ahuis, T. P., Smyk, M. K., Laloux, C., Aulehner, K., Bray, J. G., Waldron, A., Miljanovic, N., Seiffert, I., Song, D., Boulanger, B., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309521 · Cited by 10
CONSORT 2025 statement: updated guideline for reporting randomized trials: a Korean translation
Hopewell, S., Chan, A., Collins, G. S., Hróbjartsson, A., Moher, D., Schulz, K. F., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · Ewha Medical Journal · doi:10.12771/emj.2025.00409 · Cited by 8
Self-citationAR-TV and AR-Diànshì: Cultural Differences in Users’ Preferences for Augmented Reality Television
Popovici, I., Vatavu, R.-D., Feng, P., & Wu, W. (2021) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3452918.3458801 · Cited by 9
“we present two more opportunities for future work in ARTV by capitalizing on replications and research reproducibility [10].”
MethodGalloway, A., Brunet, D., Valipour, R., McCusker, M., Biberhofer, J., Sobol, M. K., Moussa, M. A., & Taylor, G. W. (2022) · Limnology and Oceanography Methods · doi:10.1002/lom3.10483 · Cited by 10
Replicability in cancer omics data analysis: measures and empirical explorations
Wang, J., Liang, H., Zhang, Q., & Ma, S. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac304 · Cited by 8
Minimum standards for evaluating machine-learned models of high-dimensional data
Chen, B. H. (2022) · Frontiers in Aging · doi:10.3389/fragi.2022.901841 · Cited by 9
“Moreover, leveraging a differential privacy framework can preserve anonymity of individual data by introducing a predetermined amount of “fake data,” which can then be accounted for when training models (Heil et al., 2021).”
BackgroundMethodOne truth and one standard for its telling: Reporting on and about scientific business research
Joullié, J., & Gould, A. M. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2022.113565 · Cited by 13
Kianersi, S., Grant, S., Naaman, K., Henschel, B., Mellor, D. T., Apte, S., Deyoe, J. E., Eze, P., Huo, C., Lavender, B. L., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221149735 · Cited by 14
“We initially identified eight clearinghouses, from which we identified the 339 eligible journals in our sample (Grant et al., 2022; Mayo-Wilson, Grant, & Supplee, 2022).”
BackgroundFerreira, M. Y., Mitre, L. P., Bishay, A. E., Batista, S., Palavani, L. B., Oliveira, L. D. B., Semione, G. D. S., Andreão, F. F., Porto, S., Sousa, M. P., et al. (2024) · Neurosurgical Review · doi:10.1007/s10143-024-02422-z · Cited by 9
“This involves considering data synthesis, reproducibility, comparability, and bias reduction [69–71].”
Replication Redux: The Reproducibility Crisis and the Case of Deworming
Ozier, O. W. (2020) · The World Bank Research Observer · doi:10.1093/wbro/lkaa005 · Cited by 11
Kubalski, P., & Hering, D. (2020) · Journal of Hypertension · doi:10.1097/hjh.0000000000002416 · Cited by 10
Wang, X., Qin, J., Zhu, R., Zhang, S., Tian, S., Sun, Y., Wang, Q., Zhao, P., Tang, H., Wang, L., et al. (2021) · Brain Connectivity · doi:10.1089/brain.2021.0153 · Cited by 9
Metadata
- Authors
- Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Steven N.
- Venue
- Science Translational Medicine
- Published
- 2016-06
- Volume
- 8
- Issue
- 341
- Pages
- 341ps12-341ps12
- ISSN
- 1946-6234 (ISSN-L)
- 1946-6242
- OpenAlex
- W2414283705
- Cited by
- 1,293
- References
- 41
- Citations by year
- 2026: 85 · 2025: 93 · 2024: 146 · 2023: 110 · 2022: 174 · 2021: 151 · 2020: 168 · 2019: 140 · 2018: 112 · 2017: 83 · 2016: 30
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science Translational Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- John and Laura Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- research reproducibility, terminology, conceptual framework, truth in science
- MeSH terms
- Reproducibility of Results, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.