Policy: NIH plans to enhance reproducibility
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is currently an absence of a culture of enhancing standards for environmental control, monitoring, and reporting in biomedical research.
“The task of enhancing standards for environmental control, monitoring, and reporting in biomedical research may initially seem overwhelming provided the current absence of a culture to this end (Collins and Tabak, 2014; Baker, 2016; Hunter, 2017).”
From Toward Best Practices for Controlling Mammalian Cell Culture Environments (2022) · cites this paper as Collins and Tabak, 2014
The Abstract says basic experimental-design details often go unreported, and PRECLINICAL PROBLEMS identifies different lab environments as a cause of irreproducible results. The paper calls for changes across the research community, but it does not specifically describe an absence of a culture of environmental control, monitoring, and reporting. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredInability to reproduce study results, often inherent in observations from academic laboratories, is usually uncovered with relevant delay, e.g., when potential therapies transition from preclinical testing to clinical trials.
“The inability to reproduce study results, often inherent in observations from academic laboratories, are usually uncovered not without relevant delay, e.g. when potential therapies that are based on these findings transition from preclinical testing to the far more stringent conditions of clinical trials (Collins and Tabak, 2014).”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Collins and Tabak, 2014
The PRECLINICAL PROBLEMS section says preclinical research is particularly susceptible to irreproducible results, while clinical trials have more rigorous design and oversight. The PROPOSED NIH ACTIONS section discusses independently validating questionable animal-model findings before a clinical trial. Neither section says failures are usually discovered only after a substantial delay, when therapies enter clinical trials. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
Moher, D., Jones, A., Lepage, L., & for the CONSORT Group (2001) · JAMA · doi:10.1001/jama.285.15.1992 · Cited by 1,027 · see citing works
The secret lives of experiments: Methods reporting in the fMRI literature
Carp, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.07.004 · Cited by 473 · see citing works
McNutt, M. (2014) · Science · doi:10.1126/science.1250475 · Cited by 291 · see citing works
Vasilevsky, N. A., Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., LaRocca, G. M., & Haendel, M. A. (2013) · PeerJ · doi:10.7717/peerj.148 · Cited by 271 · see citing works
Announcement: Reducing our irreproducibility
(2013) · Nature · doi:10.1038/496398a · Cited by 243 · see citing works
Changing Incentives to Publish
Franzoni, C., Scellato, G., & Stephan, P. E. (2011) · Science · doi:10.1126/science.1197286 · Cited by 235 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [2]The Economist ‘Trouble at the Lab’ 2013 10 19 available at http://go.nature.com/dstij3.
- [3]US Department of Health and Human Services Office of Research Integrity Annual Report 2011 2011 US HHS 2011 available at http://go.nature.com/t7ykcv
Cited by
Sorted by citations
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660 · see citing works
The “All of Us” Research Program
The All of Us Research Program Investigators (2019) · New England Journal of Medicine · doi:10.1056/nejmsr1809937 · Cited by 2,440 · see citing works
Sex and Gender Differences in Risk, Pathophysiology and Complications of Type 2 Diabetes Mellitus
Kautzky-Willer, A., Harreiter, J., & Pacini, G. (2016) · Endocrine Reviews · doi:10.1210/er.2015-1137 · Cited by 2,043 · see citing works
Policy: NIH to balance sex in cell and animal studies
Clayton, J. A., & Collins, F. S. (2014) · Nature · doi:10.1038/509282a · Cited by 1,816 · see citing works
Self-citationScanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · see citing works
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · see citing works
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Clinical Chemistry · doi:10.1373/clinchem.2015.246280 · Cited by 984 · see citing works
Muratov, E. N., Bajorath, J., Sheridan, R. P., Tetko, I. V., Filimonov, D., Poroikov, V., Oprea, T. I., Baskin, I. I., Varnek, A., Roitberg, A., et al. (2020) · Chemical Society Reviews · doi:10.1039/d0cs00098a · Cited by 913 · see citing works
Rescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817 · see citing works
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · see citing works
Reproducibility: Standardize antibodies used in research
Bradbury, A., & Plückthun, A. (2015) · Nature · doi:10.1038/518027a · Cited by 647 · see citing works
Next-generation probes, particles, and proteins for neural interfacing
Rivnay, J., Wang, H., Fenno, L., Deisseroth, K., & Malliaras, G. G. (2017) · Science Advances · doi:10.1126/sciadv.1601649 · Cited by 514 · see citing works
Meta-research: Evaluation and Improvement of Research Methods and Practices
Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002264 · Cited by 495 · see citing works
Studying both sexes: a guiding principle for biomedicine
Clayton, J. A. (2016) · The FASEB Journal · doi:10.1096/fj.15-279554 · Cited by 358 · see citing works
Maternal immune activation and autism spectrum disorder: From rodents to nonhuman and human primates
Careaga, M., Murai, T., & Bauman, M. D. (2017) · Biological Psychiatry · doi:10.1016/j.biopsych.2016.10.020 · Cited by 348 · see citing works
FSH blockade improves cognition in mice with Alzheimer’s disease
Xiong, J., Kang, S. S., Wang, Z., Liu, X., Kuo, T.-C., Korkmaz, F., Padilla, A., Miyashita, S., Chan, P., Zhang, Z., et al. (2022) · Nature · doi:10.1038/s41586-022-04463-0 · Cited by 329 · see citing works
The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications
Bik, E. M., Casadevall, A., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.00809-16 · Cited by 287 · see citing works
“The quality and integrity of the scientific literature has been increasingly questioned (8, 9, 15, 16).”
BackgroundUsing the mouse to model human disease: increasing validity and reproducibility
Justice, M. J., & Dhillon, P. (2016) · Disease Models & Mechanisms · doi:10.1242/dmm.024547 · Cited by 281 · see citing works
Artificial Intelligence for Drug Discovery: Are We There Yet?
Hasselgren, C., & Oprea, T. I. (2024) · Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-040323-040828 · Cited by 236 · see citing works
Nicholls, S. G., Quach, P., von Elm, E., Guttmann, A., Moher, D., Petersen, I., Sørensen, H. T., Smeeth, L., Langan, S. M., & Benchimol, E. I. (2015) · PLOS ONE · doi:10.1371/journal.pone.0125620 · Cited by 217 · see citing works
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215 · see citing works
Circulating metabolites and general cognitive ability and dementia: evidence from 11 cohort studies
van der Lee, S. J., Teunissen, C. E., Pool, R., Shipley, M. J., Teumer, A., Chouraki, V., Melo van Lent, D., Tynkkynen, J., Fischer, K., Hernesniemi, J., et al. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2017.11.012 · Cited by 197 · see citing works
Is animal research sufficiently evidence based to be a cornerstone of biomedical research?
Pound, P., & Bracken, M. B. (2014) · BMJ · doi:10.1136/bmj.g3387 · Cited by 207 · see citing works
In Vitro research Reproducibility: Keeping up High Standards
Hirsch, C., & Schildknecht, S. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01484 · Cited by 161 · see citing works
Reproducible and reusable research: are journal data sharing policies meeting the mark?
Vasilevsky, N. A., Minnier, J., Haendel, M. A., & Champieux, R. E. (2017) · PeerJ · doi:10.7717/peerj.3208 · Cited by 165 · see citing works
PIC, a paediatric-specific intensive care database
Zeng, X., Yu, G., Lu, Y., Tan, L., Wu, X., Shi, S., Duan, H., Shu, Q., & Li, H. (2020) · Scientific Data · doi:10.1038/s41597-020-0355-4 · Cited by 138 · see citing works
Reproducibility: changing the policies and culture of cell line authentication
Freedman, L. P., Gibson, M. C., Ethier, S. P., Soule, H. R., Neve, R. M., & Reid, Y. A. (2015) · Nature Methods · doi:10.1038/nmeth.3403 · Cited by 136 · see citing works
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138 · see citing works
Early life stress in rodents: Animal models of illness or resilience?
Murthy, S., & Gould, E. (2018) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2018.00157 · Cited by 120 · see citing works
Bowen, A., & Casadevall, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1504955112 · Cited by 104 · see citing works
Reproducibility2020: Progress and priorities
Freedman, L. P., Venugopalan, G., & Wisman, R. (2017) · F1000Research · doi:10.12688/f1000research.11334.1 · Cited by 102 · see citing works
Opportunities and challenges in combination gene cancer therapy
Nastiuk, K. L., & Krolewski, J. J. (2016) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2015.12.005 · Cited by 85 · see citing works
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83 · see citing works
Digital open science—Teaching digital tools for reproducible and transparent research
Toelch, U., & Ostwald, D. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006022 · Cited by 80 · see citing works
Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research
Ioannidis, J. P. A. (2017) · JAMA · doi:10.1001/jama.2017.0549 · Cited by 78 · see citing works
Toward good in vitro reporting standards
Hartung, T. (2019) · ALTEX · doi:10.14573/altex.1812191 · Cited by 63 · see citing works
“Rennie and Flanagin, 2018 Brazma et al., 2001 Eskes et al., 2017 Landis et al., 2012; Collins and Tabak, 2014; Clayton and Collins, 2014; Lorsch et al., 2014 O’Brien et al., 2014 Nakayama et al., 2005 Foster and Deardorff, 2017 ahttps://bit.ly/1F0K2O6; bhttps://www.nlm.nih.gov/cde/index.html;…”
BackgroundBiomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61 · see citing works
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49 · see citing works
Wayant, C., Page, M. J., & Vassar, M. (2019) · JAMA Oncology · doi:10.1001/jamaoncol.2019.2564 · Cited by 46 · see citing works
Goals in Nutrition Science 2015–2020
Allison, D. B., Bassaganya-Riera, J., Burlingame, B., Brown, A. W., le Coutre, J., Dickson, S. L., van Eden, W., Garssen, J., Hontecillas, R., Khoo, C. S. H., et al. (2015) · Frontiers in Nutrition · doi:10.3389/fnut.2015.00026 · Cited by 45 · see citing works
Boulbes, D. R., Costello, T., Baggerly, K., Fan, F., Wang, R., Bhattacharya, R., Ye, X., & Ellis, L. M. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-18-0227 · Cited by 41 · see citing works
Toward assessing clinical trial publications for reporting transparency
Kilicoglu, H., Rosemblat, G., Hoang, L., Wadhwa, S., Peng, Z., Malički, M., Schneider, J., & ter Riet, G. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103717 · Cited by 34 · see citing works
The opportunities of mining historical and collective data in drug discovery
Wassermann, A. M., Lounkine, E., Davies, J. W., Glick, M., & Camargo, L. M. (2015) · Drug Discovery Today · doi:10.1016/j.drudis.2014.11.004 · Cited by 34 · see citing works
Evidence appraisal: a scoping review, conceptual framework, and research agenda
Goldstein, A., Venker, E., & Weng, C. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx050 · Cited by 30 · see citing works
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28 · see citing works
Trends and predictors of biomedical research quality, 1990-2015: a meta-research study
Catillon, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030342 · Cited by 27 · see citing works
A New Account of Replication in the Experimental Life Sciences
Guttinger, S. (2019) · Philosophy of Science · doi:10.1086/703555 · Cited by 27 · see citing works
Hunniford, V. T., Grudniewicz, A., Fergusson, D. A., Montroy, J., Grigor, E., Lansdell, C., Lalu, M. M., & On behalf of The Canadian Critical Care Translational Biology Group (2023) · eLife · doi:10.7554/elife.76300 · Cited by 26 · see citing works
Examining the predictive validity of NIH peer review scores
Lindner, M. D., & Nakamura, R. K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126938 · Cited by 24 · see citing works
Toward Best Practices for Controlling Mammalian Cell Culture Environments
Klein, S. G., Steckbauer, A., Alsolami, S. M., Arossa, S., Parry, A. J., Li, M., & Duarte, C. M. (2022) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2022.788808 · Cited by 23 · see citing works
Franco, C. M. D. C., Carneiro, M. A. D. S., Alves, L. T. H., Júnior, G. N. D. O., de Sousa, J. D. F. R., & Orsatti, F. L. (2019) · Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000002970 · Cited by 21 · see citing works
Georgescu, C., & Wren, J. D. (2018) · Bioinformatics · doi:10.1093/bioinformatics/btx811 · Cited by 20 · see citing works
Better reporting for better research: a checklist for reproducibility
Kenall, A., Edmunds, S., Goodman, L., Bal, L., Flintoft, L., Shanahan, D. R., & Shipley, T. (2015) · GigaScience · doi:10.1186/s13742-015-0071-8 · Cited by 15 · see citing works
Johnson, A. L., Anderson, J. M., Bouvette, M., Pinero, I., Rauh, S., Johnson, B., Kee, M., Heigle, B., Tricco, A. C., Page, M. J., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.009 · Cited by 11 · see citing works
Woolf-King, S. E., Firkey, M., Foley, J. D., Bricker, J., Hahn, J. A., Asiago-Reddy, E., Wikier, J., Moskal, D., Sheinfil, A. Z., Ramos, J., et al. (2022) · AIDS and Behavior · doi:10.1007/s10461-022-03649-x · Cited by 10 · see citing works
- Authors
- Collins, F. S., & Tabak, L. A.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Tabak, Lawrence A.
- Venue
- Nature
- Published
- 2014-01-27
- Volume
- 505
- Issue
- 7485
- Pages
- 612-613
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC4058759
- OpenAlex
- W2048060485
- Cited by
- 1,247
- References
- 8
- Citations by year
- 2026: 23 · 2025: 36 · 2024: 48 · 2023: 86 · 2022: 62 · 2021: 105 · 2020: 91 · 2019: 117 · 2018: 124 · 2017: 148 · 2016: 167 · 2015: 154 · 2014: 82 · 2012: 1
- FWCI
- 344.69
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Institutes of Health (Z99 OD999999)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Keywords
- research reproducibility, National Institutes of Health, preclinical research, research integrity, scientific self-correction, science policy
- MeSH terms
- Animals, Female, Humans, Male, National Institutes of Health (U.S.), Pilot Projects, Research Design, Sex Characteristics, United States, Reproducibility of Results, Scientific Misconduct, Access to Information, Disclosure, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
