Overinterpretation and Misreporting of Diagnostic Accuracy Studies: Evidence of “Spin”
The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
PURPOSE: To estimate the frequency of distorted presentation and overinterpretation of results in diagnostic accuracy studies. MATERIALS AND METHODS: MEDLINE was searched for diagnostic accuracy studies published between January and June 2010 in journals with an impact factor of 4 or higher. Articles included were primary studies of the accuracy of one or more tests in which the results were compared with a clinical reference standard. Two authors scored each article independently by using a pretested data-extraction form to identify actual overinterpretation and practices that facilitate overinterpretation, such as incomplete reporting of study methods or the use of inappropriate methods (potential overinterpretation). The frequency of overinterpretation was estimated in all studies and in a subgroup of imaging studies. RESULTS: Of the 126 articles, 39 (31%; 95% confidence interval [CI]: 23, 39) contained a form of actual overinterpretation, including 29 (23%; 95% CI: 16, 30) with an overly optimistic abstract, 10 (8%; 96% CI: 3%, 13%) with a discrepancy between the study aim and conclusion, and eight with conclusions based on selected subgroups. In our analysis of potential overinterpretation, authors of 89% (95% CI: 83%, 94%) of the studies did not include a sample size calculation, 88% (95% CI: 82%, 94%) did not state a test hypothesis, and 57% (95% CI: 48%, 66%) did not report CIs of accuracy measurements. In 43% (95% CI: 34%, 52%) of studies, authors were unclear about the intended role of the test, and in 3% (95% CI: 0%, 6%) they used inappropriate statistical tests. A subgroup analysis of imaging studies showed 16 (30%; 95% CI: 17%, 43%) and 53 (100%; 95% CI: 92%, 100%) contained forms of actual and potential overinterpretation, respectively. CONCLUSION: Overinterpretation and misreporting of results in diagnostic accuracy studies is frequent in journals with high impact factors. SUPPLEMENTAL MATERIAL: http://radiology.rsna.org/lookup/suppl/doi:10.1148/radiol.12120527/-/DC1.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Sometimes study results are selectively reported, and other times researchers cannot resist unwarranted optimism in interpretation of their findings.2 3 4”
From STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies (2015) · cites this paper as [3]
In the citing paper
“Sometimes study results are selectively reported, and other times researchers cannot resist unwarranted optimism in interpretation of their findings.2 3 4”
From STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies (2015) · cites this paper as [3]
In the citing paper
“There are several ways to spin a report (4, 6, 8–10).”
From Misrepresentation and distortion of research in biomedical literature (2018) · cites this paper as [8]
In the citing paper
“Specific classifications of spin have been developed for different study designs and contexts [randomized controlled trials with nonstatistically significant results (4), observational studies evaluating an intervention (10), diagnostic accuracy studies (8), systematic reviews (9)].”
From Misrepresentation and distortion of research in biomedical literature (2018) · cites this paper as [8]
In the citing paper
“In diagnosis-accuracy studies, spin was identified in one-third of the articles published in high-impact factor journals (8), and in the field of molecular diagnostic tests, more than half of the reports overinterpreted the clinical applicability of the findings (39).”
From Misrepresentation and distortion of research in biomedical literature (2018) · cites this paper as [8]
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 4×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
BackgroundInfluentialTowards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · cited 7×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialThe STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Moher, D., Rennie, D., de Vet, H. C. W., & Lijmer, J. G. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00012-w1 · Cited by 1,124 · cited 4×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialBoutron, I. (2010) · JAMA · doi:10.1001/jama.2010.651 · Cited by 842 · cited 4×
“Longitudinal (with follow up) 17 (13) [7, 19]”
BackgroundInfluentialPepe, M. S., Feng, Z., Janes, H., Bossuyt, P. M., & Potter, J. D. (2008) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djn326 · Cited by 717 · cited 7×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialSelf-citationUnderreporting Research Is Scientific Misconduct
Chalmers, L. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100121018 · Cited by 551 · cited 4×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
BackgroundInfluentialThe Challenge of Subgroup Analyses — Reporting without Distorting
Lagakos, S. W. (2006) · New England Journal of Medicine · doi:10.1056/nejmp068070 · Cited by 504 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialWhy Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 502 · cited 5×
“Stronger conclusion in abstract 22 (17) [11–24] 9 (17) [9–30]”
BackgroundInfluentialComparative accuracy: assessing new tests against existing diagnostic pathways
Bossuyt, P. M., Irwig, L., Craig, J., & Glasziou, P. (2006) · BMJ · doi:10.1136/bmj.332.7549.1089 · Cited by 458 · cited 7×
“Stronger conclusion in abstract 22 (17) [11–24] 9 (17) [9–30]”
BackgroundInfluentialSelf-citationLeeflang, M. M. G., Moons, K. G. M., Reitsma, J. B., & Zwinderman, A. H. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096032 · Cited by 346 · cited 4×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
InfluentialSelf-citationAssessment of the accuracy of diagnostic tests: the cross-sectional study
Knottnerus, J. A., & Muris, J. W. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00206-3 · Cited by 312
Accuracy of data in abstracts of published research articles
Pitkin, R. M. (1999) · JAMA · doi:10.1001/jama.281.12.1110 · Cited by 302 · cited 3×
“overinterpretation 39 (31) [23–39] 16 (30) [17–43]”
Ioannidis, J. P. A. (2011) · JAMA · doi:10.1001/jama.2011.713 · Cited by 252 · cited 2×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Examining Diagnostic Tests: An Evidence-Based Perspective
Fritz, J. M., & Wainner, R. S. (2001) · Physical Therapy · doi:10.1093/ptj/81.9.1546 · Cited by 221 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialThe Evidence Base of Clinical Diagnosis
Ulahannan, T. (2002) · Journal of the Royal Society of Medicine · doi:10.1177/014107680209500815 · Cited by 180
Publications on diagnostic test evaluation in family medicine journals
Devillé, W. L. J. M., Bezemer, P. D., & Bouter, L. M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00144-4 · Cited by 161 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialReporting of precision of estimates for diagnostic accuracy: a review
Harper, R., & Reeves, B. (1999) · BMJ · doi:10.1136/bmj.318.7194.1322 · Cited by 147 · cited 4×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
InfluentialQuality of Reporting of Diagnostic Accuracy Studies
Smidt, N., Rutjes, A. W. S., van der Windt, D. A. W. M., Ostelo, R. W. J. G., Reitsma, J. B., Bossuyt, P. M., Bouter, L. M., & de Vet, H. C. W. (2005) · Radiology · doi:10.1148/radiol.2352040507 · Cited by 138 · cited 5×
“not reported 22/63‡ (35) [24, 48] 6/25§ (24) [10, 45]”
BackgroundInfluentialSelf-citationDiagnosis of Histoplasmosis by Antigen Detection in BAL Fluid
Hage, C. A., Davis, T. E., Fuller, D., Egan, L., Witt, J. R., Wheat, L. J., & Knox, K. S. (2009) · CHEST Journal · doi:10.1378/chest.09-1702 · Cited by 137 · cited 3×
“Different groups 30 (24) [16, 31]”
Horton, R. (1995) · BMJ · doi:10.1136/bmj.310.6985.985 · Cited by 132 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialUsers' guide to detecting misleading claims in clinical research reports
Montori, V. M., Jaeschke, R., Schünemann, H. J., Bhandari, M., Brozek, J. L., Devereaux, P. J., & Guyatt, G. H. (2004) · BMJ · doi:10.1136/bmj.329.7474.1093 · Cited by 106 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Hayen, A. D., Macaskill, P., Irwig, L., & Bossuyt, P. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.08.024 · Cited by 104 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Self-citationPost hoc choice of cut points introduced bias to diagnostic research
Ewald, B. D. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.11.025 · Cited by 103 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
"Spin" in scientific writing: scientific mischief and legal jeopardy.
Fletcher, R. H., & Black, B. (2007) · PubMed · Cited by 100
Why we need confidence intervals
Altman, D. G. (2005) · World Journal of Surgery · doi:10.1007/s00268-005-7911-0 · Cited by 88 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Fontela, P. S., Pant Pai, N., Schiller, I., Dendukuri, N., Ramsay, A., & Pai, M. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007753 · Cited by 76 · cited 2×
“overinterpretation 39 (31) [23–39] 16 (30) [17–43]”
The Evidence Base of Clinical Diagnosis
Ulahannan, T. (2002) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.95.8.422 · Cited by 73
Overinterpretation of Clinical Applicability in Molecular Diagnostic Research
Lumbreras, B., Parker, L. A., Porta, M., Pollán, M., Ioannidis, J. P. A., & Hernández-Aguado, I. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.121517 · Cited by 65 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialWilczynski, N. L. (2008) · Radiology · doi:10.1148/radiol.2483072067 · Cited by 63
Research Ethics: Ethical Issues of Data Reporting and the Quest for Authenticity
Marco, C. A., & Larkin, G. L. (2000) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2000.tb02049.x · Cited by 57 · cited 4×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialla Fougère, C., Pöpperl, G., Levin, J., Wängler, B., Böning, G., Uebleis, C., Cumming, P. D., Bartenstein, P., Bötzel, K., & Tatsch, K. (2010) · Journal of Nuclear Medicine · doi:10.2967/jnumed.109.071811 · Cited by 57 · cited 2×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Performance of Detecting IgM Antibodies against Enterovirus 71 for Early Diagnosis
Xu, F., Yan, Q., Wang, H., Niu, J., Li, L., Zhu, F., He, S., Zhang, S., Weng, Z., Cheng, T., et al. (2010) · PLoS ONE · doi:10.1371/journal.pone.0011388 · Cited by 53 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Analysis of Data on the Accuracy of Diagnostic Tests
Habbema, J. D. F., Eijkemans, R., Krijnen, P., & Knottnerus, J. A. (2008) · The Evidence Base of Clinical Diagnosis · doi:10.1002/9781444300574.ch7 · Cited by 51 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Cautionary tales in the clinical interpretation of studies of diagnostic tests
Scott, I. A., Greenberg, P. B., & Poole, P. J. (2007) · Internal Medicine Journal · doi:10.1111/j.1445-5994.2007.01436.x · Cited by 46 · cited 7×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialReporting of Effect Direction and Size in Abstracts of Systematic Reviews
Beller, E. M., Glasziou, P. P., Hopewell, S., & Altman, D. G. (2011) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.2011.1620 · Cited by 42 · cited 1×
“overinterpretation 39 (31) [23–39] 16 (30) [17–43]”
Ten Common Statistical Errors and How to Avoid Them
Zinsmeister, A. R., & Connor, J. T. (2008) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2007.01590.x · Cited by 38 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialSTARD Statement: Still Room for Improvement in the Reporting of Diagnostic Accuracy Studies
Bossuyt, P. M. (2008) · Radiology · doi:10.1148/radiol.2483080868 · Cited by 36 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Self-citationThe thin line between hope and hype in biomarker research.
Bossuyt, P. (2011) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.2011.729 · Cited by 32 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
BackgroundSelf-citationAreia, M., Soares, M., & Dinis-Ribeiro, M. (2010) · Endoscopy · doi:10.1055/s-0029-1243846 · Cited by 27
Selman, T. J., Morris, R. K., Zamora, J., & Khan, K. S. (2011) · BMC Women's Health · doi:10.1186/1472-6874-11-8 · Cited by 26 · cited 1×
“overinterpretation 39 (31) [23–39] 16 (30) [17–43]”
Designing Studies to Ensure that Estimates of Test Accuracy Will Travel
Irwig, L., Bossuyt, P. M., Glasziou, P., Gatsonis, C., & Lijmer, J. G. (2008) · Wiley eBooks · doi:10.1002/9781444300574.ch6 · Cited by 20 · cited 5×
“Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]”
BackgroundInfluentialSelf-citationThe Quality of Reporting of Diagnostic Accuracy Studies in Glaucoma Using Scanning Laser Polarimetry
Paranjothy, B., Shunmugam, M., & Azuara-Blanco, A. (2007) · Journal of Glaucoma · doi:10.1097/ijg.0b013e3180457c6d · Cited by 16 · cited 3×
“Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]”
Research works printed but not linked: 2
- Fletcher RH, 2007, Med Law, 26, 511
- Habbema JDF, 2002, The evidence base of clinical diagnosis, 117
Cited by
STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660
Self-citationSTARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570
“nostic test accuracy studies published in high-impact journals in 2010 revealed that 88% did not state a clear hypothesis.(22) In the first example, the authors’ objective was to evaluate the accuracy of three diagnostic strategies; their spe-”
Self-citationSTARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,337
Self-citationReducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319
Self-citationSTARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Clinical Chemistry · doi:10.1373/clinchem.2015.246280 · Cited by 984
“interpretation of their findings [2-4].”
Self-citation
Metadata
- Authors
- Ochodo, E. A., de Haan, M. C., Reitsma, J. B., Hooft, L., Bossuyt, P. M., & Leeflang, M. M. G.
- Affiliations
- Amsterdam UMC Location University of Amsterdam (NL)
- University Medical Center Utrecht (NL)
- University of Amsterdam (NL)
- Venue
- Radiology
- Published
- 2013-05
- Volume
- 267
- Issue
- 2
- Pages
- 581-588
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2148203699
- Cited by
- 158
- References
- 44
- Citations by year
- 2026: 9 · 2025: 5 · 2024: 8 · 2023: 9 · 2022: 13 · 2021: 14 · 2020: 13 · 2019: 21 · 2018: 10 · 2017: 14 · 2016: 8 · 2015: 14 · 2014: 14 · 2013: 5
- FWCI
- 15.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Pure Amsterdam UMC (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- overinterpretation, misreporting, diagnostic accuracy studies, research integrity, reporting quality, sample size determination, subgroup analysis, image studies, high-impact journals, publication bias, selective reporting, hypothesis testing, research misconduct
- MeSH terms
- Diagnostic Imaging, Diagnostic Tests, Routine, Humans, Periodicals as Topic, Publishing, Research Design, Confidence Intervals, Guidelines as Topic, Journalism, Medical, Guideline Adherence, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.