The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Empirical evidence of design-related bias in studies of diagnostic tests.
Abstract
CONTEXT: The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results. OBJECTIVE: To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy. DESIGN AND SETTING: Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model. MAIN OUTCOME MEASURES: Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design. RESULTS: Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7). CONCLUSION: These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite their apparent simplicity, such studies are at risk of bias.6 7”
From STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies (2015) · cites this paper as [6]
In the citing paper
“Knowledge of the predictors might influence outcome assessment, resulting in a biased estimation of the predictor effects for the outcome [6],[17],[39].”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [17]
Citations
Cites
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Users' Guides to the Medical Literature
Jaeschke, R. (1994) · JAMA · doi:10.1001/jama.271.9.703 · Cited by 2,688
Problems of Spectrum and Bias in Evaluating the Efficacy of Diagnostic Tests
Ransohoff, D. F., & Feinstein, A. R. (1978) · New England Journal of Medicine · doi:10.1056/nejm197810262991705 · Cited by 1,739
Moses, L. E., Shapiro, D. E., & Littenberg, B. (1993) · Statistics in Medicine · doi:10.1002/sim.4780121403 · Cited by 1,405
Users' Guides to the Medical Literature
Jaeschke, R. (1994) · JAMA · doi:10.1001/jama.271.5.389 · Cited by 1,174
THE ESTIMATION AND SIGNIFICANCE OF THE LOGARITHM OF A RATIO OF FREQUENCIES
HALDANE, B. J. B. S. (1956) · Annals of Human Genetics · doi:10.1111/j.1469-1809.1955.tb01285.x · Cited by 1,083
Selection and interpretation of diagnostic tests and procedures. Principles and applications.
Griner, P. F., Mayewski, R. J., Mushlin, A. I., & Greenland, P. (1981) · PubMed · Cited by 765
Use of Methodological Standards in Diagnostic Test Research
Reid, M. C. (1995) · JAMA · doi:10.1001/jama.274.8.645 · Cited by 710
Assessment of diagnostic tests when disease verification is subject to selection bias
Begg, C. B., & Greenes, R. A. (1983) · Biometrics · doi:10.2307/2530820 · Cited by 708
Meta-analysis of Pap Test Accuracy
Fahey, M. T., Irwig, L., & Macaskill, P. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117485 · Cited by 690
Cited by
Sorted by citations
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660
“Despite their apparent simplicity, such studies are at risk of bias [6, 7].”
BackgroundSelf-citationReitsma, J. B., Glas, A. S., Rutjes, A. W. S., Scholten, R. J. P. M., Bossuyt, P. M., & Zwinderman, A. H. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.02.022 · Cited by 3,369
“Again, the effect of these differences in design and conduct will be estimated the sROC approach as changes on the diagnostic odds scale[8,29].”
MethodSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,270
Metadata
- Authors
- Lijmer, J. G.
- Affiliations
- Amsterdam UMC Location University of Amsterdam (NL)
- Corresponding author
- Lijmer, Jeroen G.
- ORCID
- Lijmer, Jeroen G.
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 1999-09-15
- Volume
- 282
- Issue
- 11
- Pages
- 1061
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W1971348077
- Cited by
- 1,861
- References
- 38
- Citations by year
- 2026: 56 · 2025: 38 · 2024: 38 · 2023: 45 · 2022: 45 · 2021: 48 · 2020: 74 · 2019: 50 · 2018: 73 · 2017: 66 · 2016: 79 · 2015: 82 · 2014: 73 · 2013: 78 · 2012: 79
- FWCI
- 92.03
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JAMA (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Pure Amsterdam UMC (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- diagnostic accuracy, study design bias, methodological quality, diagnostic odds ratio, reference materials
- MeSH terms
- Diagnostic Tests, Routine, Evaluation Studies as Topic, Quality Control, Reference Standards, Regression Analysis, Research Design, Sensitivity and Specificity, Bias, Odds Ratio
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
