Reproducibility
Abstract
Science advances on a foundation of trusted discoveries. Reproducing an experiment is one important approach that scientists use to gain confidence in their conclusions. Recently, the scientific community was shaken by reports that a troubling proportion of peer-reviewed preclinical studies are not reproducible. Because confidence in results is of paramount importance to the broad scientific community, we are announcing new initiatives to increase confidence in the studies published in Science . For preclinical studies (one of the targets of recent concern), we will be adopting recommendations of the U.S. National Institute of Neurological Disorders and Stroke (NINDS) for increasing transparency. * Authors will indicate whether there was a pre-experimental plan for data handling (such as how to deal with outliers), whether they conducted a sample size estimation to ensure a sufficient signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter was blind to the conduct of the experiment. These criteria will be included in our author guidelines.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The original studies may have inflated effect sizes due to publication, selection, reporting, or other biases.
“An important qualification about this result is the possibility that the original studies have inflated effect sizes due to publication, selection, reporting, or other biases ( 9, 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [12]
There is little direct evidence about the rate and predictors of reproducibility.
“There is plenty of concern ( 913 ), but little direct evidence about the rate and predictors of reproducibility.”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [12]
Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results.
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [12]
Concerns about reproducibility are widespread across disciplines.
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [12]
Direct replication is rarely carried out in empirical social science.
“While direct replication of this type is widely applauded ( 1 ), it is rarely carried out in empirical social science.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [1]
Cited by
Sorted by citations
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · see citing works
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247 · see citing works
Donoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
Poulton, R., Moffitt, T. E., & Silva, P. A. (2015) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-015-1048-8 · Cited by 394 · see citing works
FSH blockade improves cognition in mice with Alzheimer’s disease
Xiong, J., Kang, S. S., Wang, Z., Liu, X., Kuo, T.-C., Korkmaz, F., Padilla, A., Miyashita, S., Chan, P., Zhang, Z., et al. (2022) · Nature · doi:10.1038/s41586-022-04463-0 · Cited by 329 · see citing works
O. B. O. T. C.-S. G., Grant, S., Mayo-Wilson, E., Montgomery, P., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2735-z · Cited by 348 · see citing works
on behalf of the CONSORT-SPI Group, Montgomery, P., Grant, S., Mayo-Wilson, E., Macdonald, G. M., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2733-1 · Cited by 333 · see citing works
A simplified guide to randomized controlled trials
Bhidé, A., Shah, P. S., & Acharya, G. (2018) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13309 · Cited by 306 · see citing works
The readability of scientific texts is decreasing over time
Plavén‐Sigray, P., Matheson, G. J., Schiffler, B. C., & Thompson, W. H. (2017) · eLife · doi:10.7554/elife.27725 · Cited by 259 · see citing works
Bon-EV: an improved multiple testing procedure for controlling false discovery rates
Li, D., Xie, Z., Zand, M. S., Fogg, T. T., & Dye, T. (2017) · BMC Bioinformatics · doi:10.1186/s12859-016-1414-x · Cited by 231 · see citing works
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215 · see citing works
Practical Computational Reproducibility in the Life Sciences
Grüning, B. A., Chilton, J. M., Köster, J., et al. (2017) · Cell Systems · doi:10.1016/j.cels.2018.03.014 · Cited by 158 · see citing works
Reproducibility of computational workflows is automated using continuous analysis
Beaulieu‐Jones, B. K., & Greene, C. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3780 · Cited by 151 · see citing works
On the Reproducibility of Psychological Science
Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1240079 · Cited by 150 · see citing works
Uses and Reuses of Scientific Data: The Data Creators’ Advantage
Pasquetto, I. V., Borgman, C. L., & Wofford, M. F. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.fc14bf2d · Cited by 147 · see citing works
von Kortzfleisch, V. T., Karp, N. A., Palme, R., Kaiser, S., Sachser, N., & Richter, S. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-73503-4 · Cited by 137 · see citing works
Han, S., Olonisakin, T. F., Pribis, J. P., Zupetic, J., Yoon, J. H., Holleran, K. M., Jeong, K., Shaikh, N., Rubio, D. M., & Lee, J. S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0183591 · Cited by 131 · see citing works
Kidon, M. I., Blanca‐López, N., Gomes, E. R., et al. (2018) · Pediatric Allergy and Immunology · doi:10.1111/pai.12915 · Cited by 133 · see citing works
Irreproducibility in hydrogen storage material research
Broom, D. P., & Hirscher, M. (2016) · Energy & Environmental Science · doi:10.1039/c6ee01435f · Cited by 130 · see citing works
Reproducibility and Research Integrity
Resnik, D. B., & Shamoo, A. E. (2016) · Accountability in Research · doi:10.1080/08989621.2016.1257387 · Cited by 129 · see citing works
Rigor and reproducibility in rodent behavioral research
Gulinello, M. E., Mitchell, H. A., Chang, Q., O’Brien, W. T., Zhou, Z., Abel, T. G., Wang, L., Corbin, J. G., Veeraragavan, S., Samaco, R. C., et al. (2018) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2018.01.001 · Cited by 116 · see citing works
“Approaches to improve rigor, transparency and reproducibility of data are now in place in many venues (Collins & Tabak, 2014; Lithgow, Driscoll, & Phillips, 2017; McNutt, 2014a, 2014b; Moher, Simera, Schulz, Hoey, & Altman, 2008).”
Replication studies in economics—How many and which papers are chosen for replication, and why?
Mueller‐Langer, F., Fecher, B., Harhoff, D., & Wagner, G. G. (2018) · Research Policy · doi:10.1016/j.respol.2018.07.019 · Cited by 118 · see citing works
“However, in times of increasing retractions and frequent instances of scientific fraud and misconduct, scientific quality assurance mechanisms are subject to a high level of scrutiny (McNutt, 2014; Steen et al., 2013; Lacetera and Zirulia, 2011; Cokol et al., 2008).”
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · doi:10.31235/osf.io/4hmb6 · Cited by 117 · see citing works
FIGL1 and its novel partner FLIP form a conserved complex that regulates homologous recombination
Fernandes, J. B., Duhamel, M., Seguéla-Arnaud, M., Froger, N., Girard, C., Choinard, S., Solier, V., De Winne, N., De Jaeger, G., Gevaert, K., et al. (2018) · PLOS Genetics · doi:10.1371/journal.pgen.1007317 · Cited by 116 · see citing works
Biomarkers of Pathological Dissociation: A Systematic Review
Roydeva, M. I., & Reinders, A. A. T. S. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.11.019 · Cited by 116 · see citing works
Tackling COVID-19 Through Responsible AI Innovation: Five Steps in the Right Direction
Leslie, D. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4bb9d7a7 · Cited by 95 · see citing works
Reproducibility of clinical research in critical care: a scoping review
Niven, D. J., McCormick, T. J., Straus, S. E., Hemmelgarn, B. R., Jeffs, L. P., Barnes, T., & Stelfox, H. T. (2018) · BMC Medicine · doi:10.1186/s12916-018-1018-6 · Cited by 87 · see citing works
The visual white matter: The application of diffusion MRI and fiber tractography to vision science
Rokem, A., Takemura, H., Bock, A. S., Scherf, K. S., Behrmann, M., Wandell, B. A., Fine, I., Bridge, H., & Pestilli, F. (2017) · Journal of Vision · doi:10.1167/17.2.4 · Cited by 84 · see citing works
An investigation of irreproducibility in maximum likelihood phylogenetic inference
Shen, X., Li, Y., Hittinger, C. T., Chen, X., & Rokas, A. (2020) · Nature Communications · doi:10.1038/s41467-020-20005-6 · Cited by 84 · see citing works
Effect of impact factor and discipline on journal data sharing policies
Resnik, D. B., Morales, M., Landrum, R. E., Shi, M., Minnier, J., Vasilevsky, N., & Champieux, R. (2019) · Accountability in Research · doi:10.1080/08989621.2019.1591277 · Cited by 82 · see citing works
“In response, funding agencies, professional societies, and journals have developed or refined policies to promote transparency, including the creation of data sharing policies (Collins and Tabak 2014; McNutt 2014; Vasilevsky et al. 2017; Stuart et al. 2018).”
BackgroundReproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77 · see citing works
brainlife.io: a decentralized and open-source cloud platform to support neuroscience research
Hayashi, S., Caron, B. A., Heinsfeld, A. S., Vinci‐Booher, S., McPherson, B., Bullock, D. N., Bertò, G., Niso, G., Hanekamp, S., Levitas, D. J., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02237-2 · Cited by 77 · see citing works
Ryan, M., Stucky, M., Wakefield, C., et al. (2020) · F1000Research · doi:10.12688/f1000research.20590.2 · Cited by 75 · see citing works
The researchers’ view of scientific rigor—survey on the conduct and reporting of in vivo research
Reichlin, T. S., Vogt, L., & Würbel, H. (2016) · PLOS ONE · doi:10.1371/journal.pone.0165999 · Cited by 72 · see citing works
Fair game? Producing and publishing gambling research
Cassidy, R. (2014) · International Gambling Studies · doi:10.1080/14459795.2014.971420 · Cited by 67 · see citing works
Research Data Sharing: Practices and Attitudes of Geophysicists
Tenopir, C., Christian, L., Allard, S. L., & Borycz, J. (2018) · Earth and Space Science · doi:10.1029/2018ea000461 · Cited by 66 · see citing works
“…(Putri et al., 2015; van den Van Den Eynden et al., 2016; Zuiderwijk & Janssen, 2014), reproducibility of science has become a growing concern (McNutt, 2014; Yaffe & Koch, 2015), and scientists increasingly recognize that there are real benefits to open data (Lowndes et al., 2017; McKiernan…”
BackgroundMore Than One Replication Study Is Needed for Unambiguous Tests of Replication
Hedges, L. V., & Schauer, J. M. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619852953 · Cited by 64 · see citing works
López‐Nicolás, R., López-López, J. A., Rubio‐Aparicio, M., & Sánchez‐Meca, J. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01644-z · Cited by 66 · see citing works
“The past decade has revealed significant problems in terms of replicability and reproducibility in psychological research, leading to the so-called replication crisis (McNutt, 2014; Open Science Collaboration, 2015; Pashler &Wagenmakers, 2012).”
BackgroundRyan, M., Stucky, M., Wakefield, C., et al. (2019) · F1000Research · doi:10.12688/f1000research.20590.1 · Cited by 63 · see citing works
Challenges in Building Usable Knowledge in Education
Hedges, L. V. (2017) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1375583 · Cited by 63 · see citing works
Surveying 5G Techno-Economic Research to Inform the Evaluation of 6G Wireless Technologies
Oughton, E. J., & Lehr, W. H. (2022) · IEEE Access · doi:10.1109/access.2022.3153046 · Cited by 62 · see citing works
User-friendly workflows for catchment modelling: Towards reproducible SWAT+ model studies
Chawanda, C. J., George, C., Thiery, W., van Griensven, A., Tech, J., Arnold, J. G., & Srinivasan, R. (2020) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104812 · Cited by 60 · see citing works
“It has been discovered that a large proportion of scientific research is not reproducible (Vasilevsky et al., 2013; McNutt, 2014).”
BackgroundTransparency and Openness Promotion (TOP) GuidelinesPreprint
Nosek, B. A., Alter, G. C., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S., Buck, S., Chambers, C., Chin, G. J., Christensen, G., et al. (2016) · doi:10.31219/osf.io/vj54c · Cited by 60 · see citing works
Self-citationRigor or mortis: best practices for preclinical research in neuroscience.
Steward, O., & Balice-Gordon, R. (2014) · Neuron · doi:10.1016/j.neuron.2014.10.042 · Cited by 58 · see citing works
Fecher, B., Friesike, S., Hebing, M., & Linek, S. B. (2017) · Humanities and Social Sciences Communications · doi:10.1057/palcomms.2017.51 · Cited by 61 · see citing works
Changing the Mindset in Life Sciences Toward Translation: A Consensus
Duda, G. N., Grainger, D. W., Frisk, M. L., Bruckner‐Tuderman, L., Carr, A. J., Dirnagl, U., Einhäupl, K. M., Gottschalk, S. M., Gruskin, E. A., Huber, C., et al. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.aaa0599 · Cited by 55 · see citing works
Injunctive Norms and Alcohol Consumption: A Revised Conceptualization
Krieger, H., Neighbors, C., Lewis, M. A., LaBrie, J. W., Foster, D. W., & Larimer, M. E. (2016) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.13037 · Cited by 58 · see citing works
RIPOSTE: a framework for improving the design and analysis of laboratory-based research
Masca, N. G. D., Hensor, E. M. A., Cornelius, V. R., Buffa, F. M., Marriott, H. M., Eales, J., Messenger, M. P., Anderson, A. E., Boot, C., Bunce, C., et al. (2015) · eLife · doi:10.7554/elife.05519 · Cited by 57 · see citing works
Olazabal, M., Neumann, M. B., Foudi, S., & Chiabai, A. (2018) · Systems Research and Behavioral Science · doi:10.1002/sres.2519 · Cited by 56 · see citing works
Nitschke, F., McKimmie, B. M., & Vanman, E. J. (2019) · Psychological Bulletin · doi:10.1037/bul0000206 · Cited by 61 · see citing works
Permutation tests are robust and powerful at 0.5% and 5% significance levels
Noguchi, K., Konietschke, F., Marmolejo‐Ramos, F., & Pauly, M. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01595-5 · Cited by 55 · see citing works
“5 Department of Statistics, TU Dortmund University, Dortmund, 44227, Germany significance tests’ (McNutt, 2014), Johnson (2013) and”
BackgroundThe Power and the Pitfalls of Large-scale, Unreplicated Natural Experiments
Barley, S. C., & Meeuwig, J. J. (2016) · Ecosystems · doi:10.1007/s10021-016-0028-5 · Cited by 53 · see citing works
Open science, the replication crisis, and environmental public health
Hicks, D. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1962713 · Cited by 56 · see citing works
“The NPR uses this phrase explicitly, giving citations to major commentaries on the crisis including Munafò et al. (2017), Ioannidis (2005), McNutt (2014), and Goodman, Fanelli, and Ioannidis (2016).”
MethodRichardson, D. B., Kwan, M., Alter, G. C., & McKendry, J. E. (2015) · Annals of GIS · doi:10.1080/19475683.2015.1027792 · Cited by 51 · see citing works
Bibri, S. E. (2019) · Smart Cities · doi:10.3390/smartcities2020013 · Cited by 53 · see citing works
Clarifying the terminology that describes scientific reproducibility
Kenett, R. S., & Galit Shmueli, A. (2015) · Nature Methods · doi:10.1038/nmeth.3489 · Cited by 50 · see citing works
Selective Inference: The Silent Killer of Replicability
Benjamini, Y. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.fc62b261 · Cited by 49 · see citing works
Test-retest reliability of spatial navigation in adults at-risk of Alzheimer’s disease
Coughlan, G., Puthusseryppady, V., Lowry, E., Gillings, R., Spiers, H. J., Minihane, A. M., & Hornberger, M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0239077 · Cited by 46 · see citing works
Publish and Who Should Perish: You or Science?
Kun, Á. (2018) · Publications · doi:10.3390/publications6020018 · Cited by 44 · see citing works
Reproducibility of Results in Preclinical Studies: A Perspective From the Bone Field
Manolagas, S. C., & Kronenberg, H. M. (2014) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2293 · Cited by 43 · see citing works
Current Status of Single Particle Imaging with X-ray Lasers
Sun, Z., Fan, J., Li, H., & Jiang, H. (2018) · Applied Sciences · doi:10.3390/app8010132 · Cited by 41 · see citing works
FSH-blocking therapeutic for osteoporosis
Gera, S., Kuo, T., Gumerova, A. A., Korkmaz, F., Sant, D., DeMambro, V., Sudha, K., Padilla, A., Prévot, G., Munitz, J., et al. (2022) · eLife · doi:10.7554/elife.78022 · Cited by 38 · see citing works
Introduction to the Special Section on Advancing Our Methods and Practices
Ledgerwood, A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529448 · Cited by 38 · see citing works
Campbell, R., Javorka, M., Engleton, J., Fishwick, K., Gregory, K., & Goodman-Williams, R. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231205832 · Cited by 35 · see citing works
“Verifying reproducibility is critical for correcting or preventing errors in published works (McNutt, 2014).”
Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science
Sakaluk, J., Williams, A., & Biernat, M. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614549257 · Cited by 32 · see citing works
Westwood, A. R., Otto, S. P., Mooers, A. Ø., et al. (2019) · FACETS · doi:10.1139/facets-2018-0042 · Cited by 36 · see citing works
“(concluded ) Open information Standards of open data have increasingly been recognized as a best scientific practice (White et al. 2013; McNutt 2014a; Munafò et al. 2017) by the Government of Canada (Gregr et al. 2017), Canada’s three federal research granting agencies (Government of Canada 2016c),…”
BackgroundDenaxas, S., Direk, K., González-Izquierdo, A., Pikoula, M., Cakiroglu, S. A., Moore, J. H., Hemingway, H., & Smeeth, L. L. (2017) · BioData Mining · doi:10.1186/s13040-017-0151-7 · Cited by 33 · see citing works
Reliability generalization meta-analysis: comparing different statistical methods
López‐Ibáñez, C., López‐Nicolás, R., Blázquez-Rincón, D., & Sánchez‐Meca, J. (2024) · Current Psychology · doi:10.1007/s12144-023-05604-y · Cited by 33 · see citing works
“It is to be expected that, as the transparency and reproducibility principles of the Open Science are implemented in psychological research, meta-analytic databases will be more accessible (Lakens et al., 2016; McNutt, 2014; Pashler & Wagenmakers, 2012).”
The international MAQC Society launches to enhance reproducibility of high-throughput technologies
Shi, L. M., Kusko, R. L., Wolfinger, R. D., Haibe‐Kains, B., Fischer, M., Sansone, S., Mason, C. E., Furlanello, C., Jones, W. D., Ning, B., et al. (2017) · Nature Biotechnology · doi:10.1038/nbt.4029 · Cited by 31 · see citing works
Plant studies on fluctuating asymmetry in Russia: Mythology and methodology
Kozlov, M. V. (2017) · Russian Journal of Ecology · doi:10.1134/s1067413617010106 · Cited by 30 · see citing works
Open-Source tools in R for forestry and forest ecology
Atkins, J. W., Stovall, A. E. L., & Silva, C. A. (2021) · Forest Ecology and Management · doi:10.1016/j.foreco.2021.119813 · Cited by 31 · see citing works
“Reproducibility is a key feature of science (McNutt, 2014; Nosek et al. 2015) that can be limited by a reliance on proprietary software.”
BackgroundMLitB: machine learning in the browser
Meeds, E., Hendriks, R., Faraby, S. A., Bruntink, M., & Welling, M. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.11 · Cited by 34 · see citing works
Dimitrova, L. I., Fernando, V., Vissia, E. M., Nijenhuis, E. R. S., Draijer, N., & Reinders, A. A. T. S. (2020) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1705599 · Cited by 31 · see citing works
Rocha, T. B.-M., Hutz, M. H., Salatino‐Oliveira, A., Genro, J. P., Polanczyk, G. V., Sato, J. R., Wehrmeister, F. C., Barros, F. C., Menezes, A. M. B., Rohde, L. A., et al. (2015) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2015.14070896 · Cited by 27 · see citing works
Behind every great research project is great data management
Kanza, S., & Knight, N. (2022) · BMC Research Notes · doi:10.1186/s13104-022-05908-5 · Cited by 27 · see citing works
Leveraging Biospecimen Resources for Discovery or Validation of Markers for Early Cancer Detection
Schully, S. D., Carrick, D. M., Mechanic, L. E., et al. (2015) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djv012 · Cited by 27 · see citing works
Trustworthy data underpin reproducible research
Milton, M. J. T., & Possolo, A. M. (2020) · Nature Physics · doi:10.1038/s41567-019-0780-5 · Cited by 26 · see citing works
Too Much Success for Recent Groundbreaking Epigenetic Experiments
Francis, G. (2014) · Genetics · doi:10.1534/genetics.114.163998 · Cited by 26 · see citing works
Wang, J., & Kwan, M. (2020) · International Journal of Health Geographics · doi:10.1186/s12942-020-00201-9 · Cited by 26 · see citing works
Hasanzadeh, K., Kajosaari, A., Häggman, D., & Kyttä, M. (2020) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2020.101513 · Cited by 26 · see citing works
Prototyping Usable Privacy and Security Systems: Insights from Experts
Mathis, F., Vaniea, K. E., & Khamis, M. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1949134 · Cited by 27 · see citing works
“Open science and reproducibility, for example, are recognized as vital features of science across research fields and considered as a disciplinary norm and value (McNutt, 2014); however, in practice there are significant differences across research communities.”
BackgroundA structured graduated protocol with heat denatured eggs in the treatment of egg allergy
Gotesdyner, L., Zeldin, Y., Machnes‐Maayan, D., et al. (2019) · Pediatric Allergy and Immunology · doi:10.1111/pai.13115 · Cited by 25 · see citing works
From methods to monographs: fostering a culture of research quality
Crawford, D. C., Hoye, M. L., & Silberberg, S. D. (2023) · eneuro · doi:10.1523/eneuro.0247-23.2023 · Cited by 25 · see citing works
Sex as a Biological Variable in Preclinical Modeling of Blast-Related Traumatic Brain Injury
McCabe, J. T., & Tucker, L. B. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.541050 · Cited by 23 · see citing works
Ben‐Zion, Z., Spiller, T. R., Keynan, J. N., Admon, R., Levy, I., Liberzon, I., Shalev, A. Y., Hendler, T., & Harpaz‐Rotem, I. (2023) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20220271 · Cited by 24 · see citing works
Wang, J., Kim, J., & Kwan, M. (2022) · Cartography and Geographic Information Science · doi:10.1080/15230406.2022.2056510 · Cited by 24 · see citing works
Reproducibility of prediction models in health services research
Belbasis, L., & Panagiotou, O. A. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06082-4 · Cited by 21 · see citing works
Spatial and temporal replicability of meteoric and in situ10Be concentrations in fluvial sediment
Gonzalez, V. S., Schmidt, A. H., Bierman, P. R., & Rood, D. H. (2017) · Earth Surface Processes and Landforms · doi:10.1002/esp.4205 · Cited by 22 · see citing works
A clinical pathway for the diagnosis of sesame allergy in children
Machnes‐Maayan, D., Yahia, S. H., Frizinsky, S., et al. (2022) · World Allergy Organization Journal · doi:10.1016/j.waojou.2022.100713 · Cited by 21 · see citing works
Research Misconduct in the Age of Open Science: The Case of STAP Stem Cells
Meskus, M., Marelli, L., & D’Agostino, G. A. (2017) · Science as Culture · doi:10.1080/09505431.2017.1316975 · Cited by 21 · see citing works
Software and the Scientist: Coding and Citation Practices in Geodynamics
Hwang, L., Fish, A. E., Soito, L., Smith, M., & Kellogg, L. H. (2017) · Earth and Space Science · doi:10.1002/2016ea000225 · Cited by 20 · see citing works
Propelling the paradigm shift from reductionism to systems nutrition
Kaput, J., Perozzi, G., Radonjić, M., & Virgili, F. (2017) · Genes & Nutrition · doi:10.1186/s12263-016-0549-8 · Cited by 20 · see citing works
van Brussel, S., Timmermans, M., Verkoeijen, P. P. J. L., & PAAS, F. G. W. C. (2020) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2020.101844 · Cited by 21 · see citing works
“(McNutt, 2014; Simmons et al., 2011). condition would be larger than in the four other conditions (Hypothesis 1a).”
Cearra, I., de la Parte, B. H., Moreno-Franco, D. I., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-98887-9 · Cited by 20 · see citing works
Collaborative methods to enhance reproducibility and accelerate discovery
Leins, D. A., Haase, S. B., Eslami, M., Schrier, J., & Freeman, J. (2022) · Digital Discovery · doi:10.1039/d2dd00061j · Cited by 23 · see citing works
Phylotocol: Promoting Transparency and Overcoming Bias in Phylogenetics
DeBiasse, M. B., & Ryan, J. F. (2018) · Systematic Biology · doi:10.1093/sysbio/syy090 · Cited by 20 · see citing works
- Authors
- McNutt, M.
- Affiliations
- Marcia McNutt is Editor-in-Chief of Science.
- Corresponding author
- McNutt, Marcia
- ORCID
- McNutt, Marcia
- Venue
- Science
- Published
- 2014-01-17
- Volume
- 343
- Issue
- 6168
- Pages
- 229-229
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4249665717
- Semantic Scholar
- 71080e83578d54a507e5e8bfe96b5a86fd1ac4d9
- Cited by
- 291
- References
- 0
- Citations by year
- 2026: 14 · 2025: 21 · 2024: 17 · 2023: 17 · 2022: 25 · 2021: 22 · 2020: 33 · 2019: 24 · 2018: 27 · 2017: 29 · 2016: 24 · 2015: 19 · 2014: 17
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- reproducibility, preclinical research, research transparency, author guidelines, sample size determination, blinding, data handling, outlier handling, randomization, peer review
- MeSH terms
- Humans, United States, Reproducibility of Results, Guidelines as Topic, Peer Review, Research, Disclosure, National Institute of Neurological Disorders and Stroke (U.S.), Translational Research, Biomedical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
