The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Biases in the assessment of diagnostic tests
Abstract
Diagnostic tests are traditionally characterized by simple measures of efficacy such as the sensitivity and the specificity. These measures, though widely recognized and easy to understand, are subject to definitional arbitrariness. Moreover, studies constructed to estimate the sensitivity and specificity are susceptible to a variety of biases. In this paper the various potential problems are described with reference to examples from the diagnostic literature. These difficulties have implications for the design of diagnostic test evaluations, and the choice of suitable measures of test efficacy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Hanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016
Basic principles of ROC analysis
Metz, C. E. (1978) · Seminars in Nuclear Medicine · doi:10.1016/s0001-2998(78)80014-2 · Cited by 6,026
Regression Models for Ordinal Data
McCullagh, P. (1980) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1980.tb01109.x · Cited by 4,453
REDUCTION IN MORTALITY FROM BREAST CANCER AFTER MASS SCREENING WITH MAMMOGRAPHY
Tabár, L., Gad, A., Holmberg, L. H., Ljungquist, U., Fagerberg, C. J. G., Baldetorp, L., Gröntoft, O., Lundström, B., Månson, J. C., Eklund, G., et al. (1985) · The Lancet · doi:10.1016/s0140-6736(85)92204-4 · Cited by 1,781
Problems of Spectrum and Bias in Evaluating the Efficacy of Diagnostic Tests
Ransohoff, D. F., & Feinstein, A. R. (1978) · New England Journal of Medicine · doi:10.1056/nejm197810262991705 · Cited by 1,734
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
(1982) · Cited by 1,288
Predictive Value of a Single Diagnostic Test in Unselected Populations
Vecchio, T. J. (1966) · New England Journal of Medicine · doi:10.1056/nejm196605262742104 · Cited by 941
Dorfman, D. D., & Alf, E. (1969) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(69)90019-4 · Cited by 874
Cited by
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
Zweig, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,569
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Time‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker
Heagerty, P. J., Lumley, T., & Pepe, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00337.x · Cited by 2,854
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Moher, D., Rennie, D., de Vet, H. C. W., & Lijmer, J. G. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00012-w1 · Cited by 1,124
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Clinical Chemistry · doi:10.1373/49.1.1 · Cited by 862
van Tinteren, H., Hoekstra, O. S., Smit, E. F., van den Bergh, J. H., Schreurs, A. J., Stallaert, R. A., van Velthoven, P. C., Comans, E. F., Diepenhorst, F. W., Verboom, P., et al. (2002) · The Lancet · doi:10.1016/s0140-6736(02)08352-6 · Cited by 851
Validation of an index of the quality of review articles
Oxman, A. D., & Guyatt, G. H. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90160-b · Cited by 786
Guidelines for Meta-analyses Evaluating Diagnostic Tests
Irwig, L., Tosteson, A. N. A., Gatsonis, C., Lau, J., Colditz, G., Chalmers, T. C., & Mosteller, F. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-8-199404150-00008 · Cited by 753
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
A systematic review of rapid diagnostic tests for the detection of tuberculosis infection
Dinnes, J., Deeks, J., Kunst, H., Gibson, A., Cummins, E., Waugh, N., Drobniewski, F., & Lalvani, A. (2007) · Health Technology Assessment · doi:10.3310/hta11030 · Cited by 666
Dwamena, B. A., Sonnad, S. S., Angobaldo, J. O., & Wahl, R. L. (1999) · Radiology · doi:10.1148/radiology.213.2.r99nv46530 · Cited by 649
Basic Methods for Sensitivity Analysis of Biases
GREENLAND, S. (1996) · International Journal of Epidemiology · doi:10.1093/ije/25.6.1107 · Cited by 586
Meta-analysis of screening and diagnostic tests.
Hasselblad, V., & Hedges, L. V. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.117.1.167 · Cited by 579
Uses and abuses of screening tests
Grimes, D. A., & Schulz, K. F. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07948-5 · Cited by 545
Statistical Evaluation of a Biomarker
Ray, P., Manach, Y. L., Riou, B., Houle, T. T., & Warner, D. S. (2010) · Anesthesiology · doi:10.1097/aln.0b013e3181d47604 · Cited by 503
Brito, J. P., Gionfriddo, M. R., Al Nofal, A., Boehmer, K. R., Leppin, A. L., Reading, C., Callstrom, M., Elraiyah, T. A., Prokop, L. J., Stan, M. N., et al. (2014) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2013-2928 · Cited by 488
BRENNER, H., & GEFELLER, O. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970515)16:9<981::aid-sim510>3.0.co;2-n · Cited by 488
Microscopic-Observation Drug-Susceptibility Assay for the Diagnosis of TB
Moore, D. A. J., Evans, C. A. W., Gilman, R. H., Caviedes, L., Coronel, J., Vivar, A., Sanchez, E., Piñedo, Y., Saravia, J. C., Salazar, C., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa055524 · Cited by 468
Meta-analytic methods for diagnostic test accuracy
Irwig, L., Macaskill, P., Glasziou, P., & Fahey, M. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00099-c · Cited by 467
Romagnuolo, J., Bardou, M., Rahme, E., Joseph, L., Reinhold, C., & Barkun, A. N. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-139-7-200310070-00006 · Cited by 445
Dinnes, J., Deeks, J., Kirby, J., & Roderick, P. (2005) · Health Technology Assessment · doi:10.3310/hta9120 · Cited by 438
The interpretation of diagnostic tests
Shapiro, D. E. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800203 · Cited by 383
Janda, S., Shahidi, N., Gin, K., & Swiston, J. (2011) · Heart · doi:10.1136/hrt.2010.212084 · Cited by 375
Development and validation of methods for assessing the quality of diagnostic accuracy studies
Whiting, P., Rutjes, A., Dinnes, J., Reitsma, J., Bossuyt, P., & Kleijnen, J. (2004) · Health Technology Assessment · doi:10.3310/hta8250 · Cited by 372
Evidence base of clinical diagnosis: Evaluation of diagnostic procedures
Knottnerus, J. A. (2002) · BMJ · doi:10.1136/bmj.324.7335.477 · Cited by 346
Spectrum Bias or Spectrum Effect? Subgroup Variation in Diagnostic Test Evaluation
Mulherin, S. A., & Miller, W. C. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-7-200210010-00011 · Cited by 327
McDowell, I., Kristjansson, B., Hill, G. B., & Hébert, R. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00060-7 · Cited by 307
Systematic Review: Noninvasive Testing for Chlamydia trachomatis and Neisseria gonorrhoeae
Cook, R. L., Hutchison, S. L., Østergaard, L., Braithwaite, R. S., & Ness, R. B. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-11-200506070-00010 · Cited by 300
Diagnosis of the condition of the dental pulp: a systematic review
Mejàre, I. A., Axelsson, S., Davidson, T., Frisk, F., Hakeberg, M., Kvist, T., Norlund, A., Petersson, A., Portenier, I., Sandberg, H., et al. (2012) · International Endodontic Journal · doi:10.1111/j.1365-2591.2012.02016.x · Cited by 296
The use of “overall accuracy” to evaluate the validity of screening or diagnostic tests
Alberg, A. J., Park, J. W., Hager, B. W., Brock, M. V., & Diener-West, M. (2004) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2004.30091.x · Cited by 295
A method to automate probabilistic sensitivity analyses of misclassified binary variables
Fox, M. P., Lash, T. L., & Greenland, S. (2005) · International Journal of Epidemiology · doi:10.1093/ije/dyi184 · Cited by 288
A systematic review classifies sources of bias and variation in diagnostic test accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., & Mallett, S. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.05.014 · Cited by 283
Defining asthma in epidemiological studies
Pekkanen, J., & Pearce, N. (1999) · European Respiratory Journal · doi:10.1034/j.1399-3003.1999.14d37.x · Cited by 279
A General Regression Methodology for ROC Curve Estimation
Tosteson, A. N. A., & Begg, C. B. (1988) · Medical Decision Making · doi:10.1177/0272989x8800800309 · Cited by 272
Self-citationDiagnostic methods I: sensitivity, specificity, and other measures of accuracy
van Stralen, K. J., Stel, V. S., Reitsma, J. B., Dekker, F. W., Zoccali, C., & Jager, K. J. (2009) · Kidney International · doi:10.1038/ki.2009.92 · Cited by 264
Irwig, L. (2002) · BMJ · doi:10.1136/bmj.324.7338.669 · Cited by 246
Evaluating diagnostic tests with imperfect standards
Valenstein, P. N. (1990) · American Journal of Clinical Pathology · doi:10.1093/ajcp/93.2.252 · Cited by 220
Meta-analyses including data from observational studies
O’Connor, A. M., & Sargeant, J. M. (2014) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.10.017 · Cited by 38
Metadata
- Authors
- Begg, C. B.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 1987-06
- Volume
- 6
- Issue
- 4
- Pages
- 411-423
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2032012575
- Cited by
- 496
- References
- 41
- FWCI
- 8.99
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.