Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests
Abstract
We review the principles and practical application of receiver-operating characteristic (ROC) analysis for diagnostic tests. ROC analysis can be used for diagnostic tests with outcomes measured on ordinal, interval or ratio scales. The dependence of the diagnostic sensitivity and specificity on the selected cut-off value must be considered for a full test evaluation and for test comparison. All possible combinations of sensitivity and specificity that can be achieved by changing the test's cut-off value can be summarised using a single parameter; the area under the ROC curve. The ROC technique can also be used to optimise cut-off values with regard to a given prevalence in the target population and cost ratio of false-positive and false-negative results. However, plots of optimisation parameters against the selected cut-off value provide a more-direct method for cut-off selection. Candidates for such optimisation parameters are linear combinations of sensitivity and specificity (with weights selected to reflect the decision-making situation), odds ratio, chance-corrected measures of association (e. g. kappa) and likelihood ratios. We discuss some recent developments in ROC analysis, including meta-analysis of diagnostic tests, correlated ROC curves (paired-sample design) and chance- and prevalence-corrected ROC curves.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A cut-off score of 8 out of 30 was chosen for the SLPPDS based on the fitted ROC curve and recommendations.
“a cut-off score of 8 out of 30 was chosen. This decision was based on the fitted ROC curve and recommendations (Greiner et al., 2000).”
From Developing and validating the Sierra Leone perinatal psychological distress scale through an emic-etic approach (2025) · cites this paper as Greiner et al., 2000
ROC curves and AUC are appropriate measures to evaluate prediction capability of logistic regression models.
“The receiver operating characteristic (ROC) curves and the areas under ROC curves (AUC) are appropriate measures to evaluate prediction capability of logistic regression models (Greiner et al., 2000; Hanley and McNeil, 1982; Meng, 2014; Zweig and Campbell, 1993).”
From Key Enhancements to the WFRC/MAG Four-Step Travel Demand Model (2019) · cites this paper as Greiner et al., 2000
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Measuring the Accuracy of Diagnostic Systems
Swets, J. A. (1988) · Science · doi:10.1126/science.3287615 · Cited by 10,058 · see citing works
Hanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016 · see citing works
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
ZWEIG, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,528 · see citing works
Basic principles of ROC analysis
Metz, C. E. (1978) · Seminars in Nuclear Medicine · doi:10.1016/s0001-2998(78)80014-2 · Cited by 6,058 · see citing works
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,458 · see citing works
Moses, L. E., Shapiro, D. E., & Littenberg, B. (1993) · Statistics in Medicine · doi:10.1002/sim.4780121403 · Cited by 1,405 · see citing works
Evaluating Medical Tests: Objective and Quantitative Guidelines.
Guthrie, D., & Kraemer, H. C. (1992) · Journal of the American Statistical Association · doi:10.2307/2290671 · Cited by 636 · see citing works
ESTIMATING THE PROPORTION OF TREATMENT EFFECT EXPLAINED BY A SURROGATE MARKER
Lin, D. Y., Fleming, T. R., & Gruttola, V. D. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970715)16:13 · Cited by 520 · see citing works
Epidemiologic issues in the validation of veterinary diagnostic tests
Greiner, M., & Gardner, I. A. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00114-8 · Cited by 485 · see citing works
Self-citationMeta-analytic methods for diagnostic test accuracy
Irwig, L., Macaskill, P., Glasziou, P., & Fahey, M. T. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00099-c · Cited by 467 · see citing works
Greiner, M., Sohr, D., & Göbel, P. (1995) · Journal of Immunological Methods · doi:10.1016/0022-1759(95)00121-p · Cited by 370 · see citing works
Self-citationConditional dependence between tests affects the diagnosis and surveillance of animal diseases
Gardner, I. A., Stryhn, H., Lind, P., & Collins, M. T. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00119-7 · Cited by 340 · see citing works
Evaluation of regression procedures for methods comparison studies
Línnet, K. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.3.424 · Cited by 324 · see citing works
The Robustness of the "Binormal" Assumptions Used in Fitting ROC Curves
Hanley, J. A. (1988) · Medical Decision Making · doi:10.1177/0272989x8800800308 · Cited by 308 · see citing works
A General Regression Methodology for ROC Curve Estimation
Tosteson, A. N. A., & Begg, C. B. (1988) · Medical Decision Making · doi:10.1177/0272989x8800800309 · Cited by 274 · see citing works
Henderson, A. R. (1993) · Annals of Clinical Biochemistry International Journal of Laboratory Medicine · doi:10.1177/000456329303000601 · Cited by 264 · see citing works
On the statistical analysis of ROC curves
Thompson, M. L., & Zucchini, W. (1989) · Statistics in Medicine · doi:10.1002/sim.4780081011 · Cited by 225 · see citing works
Sample size calculations in studies of test accuracy
Obuchowski, N. A. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700405 · Cited by 221 · see citing works
Venkatraman, E. S. (1996) · Biometrika · doi:10.1093/biomet/83.4.835 · Cited by 218 · see citing works
Advances in statistical methodology for the evaluation of diagnostic and laboratory tests
Campbell, G. D. (1994) · Statistics in Medicine · doi:10.1002/sim.4780130513 · Cited by 214 · see citing works
Slopes of a Receiver Operating Characteristic Curve and Likelihood Ratios for a Diagnostic Test
Choi, B. C. K. (1998) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009592 · Cited by 199 · see citing works
The Area under the ROC Curve and Its Competitors
Hilden, J. (1991) · Medical Decision Making · doi:10.1177/0272989x9101100204 · Cited by 183 · see citing works
Maximum Likelihood Estimation of Parameters of Signal Detection Theory–A Direct Solution
Dorfman, D. D., & Alf, E. F. (1968) · Psychometrika · doi:10.1007/bf02289677 · Cited by 177 · see citing works
Paré, J., Hietala, S. K., & Thurmond, M. C. (1995) · Journal of Veterinary Diagnostic Investigation · doi:10.1177/104063879500700310 · Cited by 169 · see citing works
On the use and computation of likelihood ratios in clinical chemistry.
Albert, A. (1982) · Clinical Chemistry · doi:10.1093/clinchem/28.5.1113 · Cited by 152 · see citing works
Greiner, M. (1996) · Journal of Immunological Methods · doi:10.1016/0022-1759(96)00013-0 · Cited by 124 · see citing works
Self-citationAN ALTERNATIVE TO THE USE OF TWO-SIDED TESTS IN CLINICAL TRIALS
Dunnett, C. W., & Gent, M. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960830)15:16 · Cited by 110 · see citing works
On Comparisons of Sensitivity, Specificity and Predictive Value of a Number of Diagnostic Procedures
Bennett, B. M. (1972) · Biometrics · doi:10.2307/2528763 · Cited by 105 · see citing works
A computer program for non-parametric receiver operating characteristic analysis
Vida, S. (1993) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(93)90004-5 · Cited by 85 · see citing works
Likelihood ratios for continuous test results—making the clinicians' job easier or harder?
Simel, D. L., Samsa, G. P., & Matchar, D. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90012-p · Cited by 77 · see citing works
Computing Sample Size for Receiver Operating Characteristic Studies
Obuchowski, N. A. (1994) · Investigative Radiology · doi:10.1097/00004424-199402000-00020 · Cited by 75 · see citing works
Constructing a cut‐off point for a quantitative diagnostic test
Schäfer, H. J. (1989) · Statistics in Medicine · doi:10.1002/sim.4780081110 · Cited by 69 · see citing works
Confidence Intervals for the Generalized ROC Criterion
Reiser, B., & Faraggi, D. (1997) · Biometrics · doi:10.2307/2533964 · Cited by 67 · see citing works
Diagnostic tests: A statistical review
Schulzer, M. (1994) · Muscle & Nerve · doi:10.1002/mus.880170719 · Cited by 64 · see citing works
Sample size calculations in studies of test accuracy
Obuchowski, N. A. (1998) · Statistical Methods in Medical Research · doi:10.1191/096228098678080061 · Cited by 59 · see citing works
Application of Logistic Regression to the Analysis of Diagnostic Data
Knottnerus, J. A. (1992) · Medical Decision Making · doi:10.1177/0272989x9201200202 · Cited by 51 · see citing works
Notes about determining the cut-off value in enzyme linked immunosorbent assay (ELISA)
Barajas-Rojas, J. A., Riemann, H. P., & Franti, C. E. (1993) · Preventive Veterinary Medicine · doi:10.1016/0167-5877(93)90116-b · Cited by 43 · see citing works
Testing for equivalence of diagnostic tests.
Obuchowski, N. A. (1997) · American Journal of Roentgenology · doi:10.2214/ajr.168.1.8976911 · Cited by 43 · see citing works
Detilleux, J., Arendt, J., Lomba, F., & Leroy, P. D. (1999) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(99)00054-9 · Cited by 38 · see citing works
Efficient confidence bounds for ROC curves
Schäfer, H. J. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131506 · Cited by 37 · see citing works
New insights into contaminants
Kardash, S. (1993) · Canadian Medical Association Journal · Cited by 35 · see citing works
Analysis of clustered data in receiver operating characteristic studies
Beam, C. A. (1998) · Statistical Methods in Medical Research · doi:10.1177/096228029800700402 · Cited by 35 · see citing works
Remaley, A. T., Sampson, M. L., DeLeo, J. M., Remaley, N. A., Farsi, B. D., & ZWEIG, M. H. (1999) · Clinical Chemistry · doi:10.1093/clinchem/45.7.934 · Cited by 30 · see citing works
Evaluation and comparison of antibody ELISAs for serodiagnosis of bovine trypanosomosis
Greiner, M., Kumar, S., & Kyeswa, C. (1997) · Veterinary Parasitology · doi:10.1016/s0304-4017(97)00134-9 · Cited by 29 · see citing works
Self-citationBeam, C. A., & Wieand, H. S. (1991) · Biometrics · doi:10.2307/2532648 · Cited by 26 · see citing works
Log-linear and logistic modeling of dependence among diagnostic tests
Hanson, T., Johnson, W. O., & Gardner, I. A. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00120-3 · Cited by 25 · see citing works
Analysis of clustered data in receiver operating characteristic studies
Beam, C. A. (1998) · Statistical Methods in Medical Research · doi:10.1191/096228098669595619 · Cited by 24 · see citing works
ROC plots display test accuracy, but are still limited by the study design
ZWEIG, M. H. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.6.1345 · Cited by 21 · see citing works
Decision analysis in the evaluation of diagnostic tests
Smith, R. D. (1993) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.1993.203.08.1184 · Cited by 20 · see citing works
Self-citationParametric likelihoods for multiple non-fatal competing risks and death
Shen, Y., & Thall, P. F. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980515)17:9 · Cited by 17 · see citing works
A modest proposal for the consistent presentation of ROC plots in Clinical Chemistry
Henderson, A. R., & Bhayana, V. (1995) · Clinical Chemistry · doi:10.1093/clinchem/41.8.1205 · Cited by 15 · see citing works
Brenner, H., & Gefeller, O. (1994) · Methods of Information in Medicine · doi:10.1055/s-0038-1635010 · Cited by 12 · see citing works
Clinical Evaluation of Test Strategies: A Decision Analysis of Parameter Estimation
Sondik, E. J. (1982) · Clinics in Laboratory Medicine · doi:10.1016/s0272-2712(18)31015-1 · Cited by 11 · see citing works
Is there a gain from “chance-corrected” measures of diagnostic validity?
Holle, R., & Windeler, J. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00311-3 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Understanding diagnostic tests 3: receiver operating characteristic curves
Akobeng, A. K. (2007) · Acta Paediatrica · doi:10.1111/j.1651-2227.2006.00178.x · Cited by 1,882 · see citing works
Chien, K.-L., Chen, P.-C., Hsu, H.-C., Su, T.-C., Sung, F.-C., Chen, M.-F., & Lee, Y.-T. (2010) · SLEEP · doi:10.1093/sleep/33.2.177 · Cited by 350 · see citing works
Haringsma, R., Engels, G. I., Beekman, A. T. F., & Spinhoven, P. (2004) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.1130 · Cited by 299 · see citing works
Exercise habit formation in new gym members: a longitudinal study
Kaushal, N., & Rhodes, R. E. (2015) · Journal of Behavioral Medicine · doi:10.1007/s10865-015-9640-7 · Cited by 285 · see citing works
Stavropoulos, V., Alexandraki, K., & Motti‐Stefanidi, F. (2013) · Journal of Adolescence · doi:10.1016/j.adolescence.2013.03.008 · Cited by 213 · see citing works
Kim, S., Hwang, J., Xuan, J., Jung, Y. H., Cha, H.-S., & Kim, K. H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0097501 · Cited by 169 · see citing works
Atrial fibrillation prevalence, incidence and risk of stroke and all-cause death among Chinese
Chien, K.-L., Su, T.-C., Hsu, H.-C., Chang, W.-T., Chen, P.-C., Chen, M.-F., & Lee, Y.-T. (2010) · International Journal of Cardiology · doi:10.1016/j.ijcard.2008.10.045 · Cited by 146 · see citing works
Comparing the Areas Under Two Correlated ROC Curves: Parametric and Non‐Parametric Approaches
Molodianovitch, K., Faraggi, D., & Reiser, B. (2006) · Biometrical Journal · doi:10.1002/bimj.200610223 · Cited by 131 · see citing works
Isokuortti, E., Zhou, Y., Peltonen, M., Bugianesi, E., Clement, K., Bonnefont-Rousselot, D., Lacorte, J.-M., Gastaldelli, A., Schuppan, D., Schattenberg, J. M., et al. (2017) · Diabetologia · doi:10.1007/s00125-017-4340-1 · Cited by 120 · see citing works
Chien, K.-L., Hsu, H.-C., Su, T.-C., Chen, M.-F., Lee, Y.-T., & Hu, F. B. (2007) · Journal of Lipid Research · doi:10.1194/jlr.m700213-jlr200 · Cited by 109 · see citing works
Delgadillo, J., Payne, S., Gilbody, S., Godfrey, C., Gore, S., Jessop, D., & Dale, V. (2012) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2012.03.011 · Cited by 106 · see citing works
Enrique, A., Palacios, J. E., Ryan, H., & Richards, D. (2019) · Journal of Medical Internet Research · doi:10.2196/12775 · Cited by 81 · see citing works
Hansen, A. B., Hartvig, N. V., Jensen, M. S., Borch‐Johnsen, K., Lund‐Andersen, H., & Larsen, M. (2004) · Acta Ophthalmologica Scandinavica · doi:10.1111/j.1600-0420.2004.00350.x · Cited by 67 · see citing works
Mugnier, A., Mila, H., Guiraud, F., Brévaux, J., Lecarpentier, M., Martinez, C., Mariani, C., Adib-Lesaux, A., Chastant-Maillard, S., Saegerman, C., et al. (2019) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2019.104746 · Cited by 70 · see citing works
Liang, C.-H., Liu, Y.-C., Wu, M.-T., Garcia-Castro, F., Alberich-Bayarri, A., & Wu, F.-Z. (2020) · Clinical Radiology · doi:10.1016/j.crad.2019.08.005 · Cited by 69 · see citing works
Moon, K.-J., Jin, Y., Jin, T., & Lee, S.-M. (2018) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.09.014 · Cited by 45 · see citing works
Ho, P.-S., Yen, C.-H., Chen, C.-Y., Huang, S.-Y., & Liang, C.-S. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2016.12.014 · Cited by 39 · see citing works
Abeyasinghe, D. R. R., Tennakoon, S., & Rajapakse, T. N. (2012) · Journal of Affective Disorders · doi:10.1016/j.jad.2012.04.019 · Cited by 32 · see citing works
Berge, S. T., Safi, N., Medhus, A. W., Ånonsen, K., Sundhagen, J. O., Hisdal, J., & Kazmi, S. S. H. (2019) · Scandinavian Journal of Clinical and Laboratory Investigation · doi:10.1080/00365513.2019.1672084 · Cited by 21 · see citing works
Reddy, P., Rochfort, S., Read, E., Deseo, M., Jaehne, E., Van Den Buuse, M., Guthridge, K., Combs, M., Spangenberg, G., & Quinn, J. (2019) · Scientific Reports · doi:10.1038/s41598-019-45170-7 · Cited by 16 · see citing works
Chen, P., Hsu, H., Chao, Y., Nouchi, R., Wang, P., & Cheng, C. (2021) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.13691 · Cited by 14 · see citing works
Bah, A. J., Wurie, H. R., Samai, M., Horn, R., & Ager, A. (2025) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2024.100852 · Cited by 7 · see citing works
Key Enhancements to the WFRC/MAG Four-Step Travel Demand Model
University of Utah, Ewing, R., Sabouri, S., University of Utah, Park, K., University of Utah, Lyons, T., University of Utah, Tian, G., & University of Utah (2019) · doi:10.15760/trec.246 · Cited by 3 · see citing works
- Authors
- Greiner, M., Pfeiffer, D., & Smith, R. D.
- Affiliations
- Freie Universität Berlin (DE)
- Royal Veterinary College (GB)
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Greiner, M
- Venue
- Preventive Veterinary Medicine
- Publisher
- Elsevier BV
- Published
- 2000-05
- Volume
- 45
- Issue
- 1-2
- Pages
- 23-41
- ISSN
- 0167-5877 (ISSN-L)
- OpenAlex
- W2121943809
- Cited by
- 1,923
- References
- 55
- Citations by year
- 2026: 48 · 2025: 63 · 2024: 93 · 2023: 90 · 2022: 96 · 2021: 109 · 2020: 122 · 2019: 112 · 2018: 119 · 2017: 131 · 2016: 136 · 2015: 98 · 2014: 104 · 2013: 89 · 2012: 95
- FWCI
- 19.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Preventive Veterinary Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- receiver operating characteristic, diagnostic test evaluation, sensitivity and specificity, area under the ROC curve, likelihood ratios, kappa statistic, odds ratio
- MeSH terms
- Animals, Animal Diseases, Diagnostic Tests, Routine, Enzyme-Linked Immunosorbent Assay, Probability, ROC Curve, Sensitivity and Specificity, Software, Prevalence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review