Increasing value and reducing waste in research design, conduct, and analysis
Open accessThe DOI matches its Crossref record; 104 of the 105 references it cites were found in a registry or an index.
Abstract
Correctable weaknesses in the design, conduct, and analysis of biomedical and public health research studies can produce misleading results and waste valuable resources. Small effects can be difficult to distinguish from bias introduced by study design and analyses. An absence of detailed written protocols and poor documentation of research is common. Information obtained might not be useful or important, and statistical precision or power is often too low or used in a misleading way. Insufficient consideration might be given to both previous and continuing studies. Arbitrary choice of analyses and an overemphasis on random extremes might affect the reported findings. Several problems relate to the research workforce, including failure to involve experienced statisticians and methodologists, failure to train clinical researchers and laboratory scientists in research methods and design, and the involvement of stakeholders with conflicts of interest. Inadequate emphasis is placed on recording of research decisions and on reproducibility of research. Finally, reward systems incentivise quantity more than quality, and novelty more than reliability. We propose potential solutions for these problems, including improvements in protocols and documentation, consideration of evidence from studies in progress, standardisation of research efforts, optimisation and training of an experienced and non-conflicted scientific workforce, and reconsideration of scientific reward systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Various calculations estimate that tens of billions of US dollars are wasted annually owing to lack of reproducibility in preclinical research25,26.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Registration does not guarantee that trial results will be ultimately published, but it is a key factor in reducing waste in research (Ioannidis et al. 2014).”
From Clinical Trial Registration in Oral Health Journals (2015) · cites this paper as Ioannidis et al. 2014
In the citing paper
“For example, the Lancet series in 2014 did not make reference to the usefulness of pilot and feasibility studies in the context of increasing value and reducing waste in research design, conduct or analysis.”
From Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study (2018) · cites this paper as [9]
In the citing paper
“Despite the importance of the role of pilot and feasibility studies in informing RCTs, there is little empirical evidence about the use of these studies in informing future trials.”
From Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study (2018) · cites this paper as [9]
In the citing paper
“The feasibility phase thus has the potential to ensure that money is not wasted on an expensive trial which produces a null result due to problems with recruitment, retention or delivery of the intervention [2].”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [2]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
Minimum information about a microarray experiment (MIAME)-toward standards for microarray data
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C. A., Causton, H. C., et al. (2001) · Nature Genetics · doi:10.1038/ng1201-365 · Cited by 4,109
Hindorff, L. A., Sethupathy, P., Junkins, H. A., Ramos, E. M., Mehta, J. P., Collins, F. S., & Manolio, T. A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0903103106 · Cited by 4,013
Modern Epidemiology 3rd edition
Rothman, K. J., Lash, T. L., & Greenland, S. (1986) · Cited by 3,096
Self-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Statistical Analysis With Missing Data
Lazar, N. A. (2003) · Technometrics · doi:10.1198/tech.2003.s167 · Cited by 2,458
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Research works printed but not linked: 1
- Greenland, 2008, Bias analysis. Chapter 19, 345
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationCOSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures
Mokkink, L. B., de Vet, H. C. W., Prinsen, C. A. C., Patrick, D. L., Alonso, J., Bouter, L. M., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-017-1765-4 · Cited by 2,479
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Self-citationHow to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
Self-citationNagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Self-citation
Metadata
- Authors
- Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R.
- Affiliations
- Stanford Medicine (US)
- University of California, Los Angeles (US)
- Centers for Disease Control and Prevention (US)
- University of Edinburgh (GB)
- University of Ottawa (CA)
- University of North Carolina at Chapel Hill (US)
- Stanford University (US)
- Corresponding author
- Ioannidis, John P A
- ORCID
- Venue
- The Lancet
- Publisher
- Elsevier BV
- Published
- 2014-01
- Volume
- 383
- Issue
- 9912
- Pages
- 166-175
- ISSN
- 0099-5355 (ISSN-L)
- 0140-6736
- PMC
- PMC4697939
- OpenAlex
- W1975130349
- Cited by
- 1,597
- References
- 105
- Citations by year
- 2026: 68 · 2025: 91 · 2024: 134 · 2023: 106 · 2022: 133 · 2021: 160 · 2020: 169 · 2019: 137 · 2018: 141 · 2017: 156 · 2016: 138 · 2015: 102 · 2014: 59
- FWCI
- 148.59
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- The Lancet (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC/L000970/1)
- National Institutes of Health (CC999999)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Nursing › Research and Theory › Methodologies in Health Research and Practice
- Keywords
- research waste, study design flaws, research bias, statistical power, research documentation, reproducibility, conflicts of interest
- MeSH terms
- Research Design, Research Personnel, Reward, Statistics as Topic, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Increasing value and reducing waste in research design, conduct, and analysis. https://app.cochat.ai/papers/10.1016/s0140-6736(13)62227-8
- The paper
- Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet. https://doi.org/10.1016/s0140-6736(13)62227-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.