Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach
Abstract
Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The DeLong test showed that GRU-D++ was significantly better than SWIFT (p < 0.001).
“The results showed that GRU-D++ was significantly better than SWIFT (p < 0.001).”
From AI-Driven Predictions of Readmission and Mortality for Improved Discharge Decisions in Critical Care: A Retrospective Study (2026) · cites this paper as [23]
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Basic principles of ROC analysis
Metz, C. E. (1978) · Seminars in Nuclear Medicine · doi:10.1016/s0001-2998(78)80014-2 · Cited by 6,058 · see citing works
A Class of Statistics with Asymptotically Normal Distribution
Hoeffding, W. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-0919-5_20 · Cited by 1,898 · see citing works
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,458 · see citing works
(1982) · Cited by 1,288 · see citing works
Dorfman, D. D., & Alf, E. F. (1969) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(69)90019-4 · Cited by 878 · see citing works
Metz, C. E., Wang, P.-L., & Kronman, H. B. (1984) · Information Processing in Medical Imaging · doi:10.1007/978-94-009-6045-9_25 · Cited by 359 · see citing works
Arvesen, J. N. (1969) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697287 · Cited by 286 · see citing works
On Some Convergence Properties of UStatistics
Sen, P. K. (1960) · Calcutta Statistical Association Bulletin · doi:10.1177/0008068319600101 · Cited by 182 · see citing works
Surgical management of bowel obstruction in advanced ovarian carcinoma.
Krebs, H. B., & Goplerud, D. R. (1983) · PubMed · Cited by 132 · see citing works
Intestinal operations in patients with ovarian carcinoma
Castaldo, T. W., Petrilli, E. S., Ballon, S. C., & Lagasse, L. D. (1981) · American Journal of Obstetrics and Gynecology · doi:10.1016/0002-9378(81)90416-6 · Cited by 91 · see citing works
Elements of Nonparametric Statistics.
Yen, E. H., & Noether, G. E. (1968) · Journal of the American Statistical Association · doi:10.2307/2284048 · Cited by 79 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Maximum entropy modeling of species geographic distributions
Phillips, S. J., Anderson, R. P., & Schapire, R. E. (2005) · Ecological Modelling · doi:10.1016/j.ecolmodel.2005.03.026 · Cited by 17,870 · see citing works
FastTree 2 – Approximately Maximum-Likelihood Trees for Large Alignments
Price, M. N., Dehal, P. S., & Arkin, A. P. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009490 · Cited by 14,894 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L. D., Beck, C. A., Feasby, T. E., & Ghali, W. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000182534.19832.83 · Cited by 10,669 · see citing works
Acute Respiratory Distress Syndrome
Ards Definition Task Force (2012) · JAMA · doi:10.1001/jama.2012.5669 · Cited by 8,806 · see citing works
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
ZWEIG, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,528 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
Levey, A. S., Coresh, J., Greene, T., Stevens, L. A., Zhang, Y., Hendriksen, S. M., Kusek, J. W., & Lente, F. V. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-4-200608150-00004 · Cited by 5,726 · see citing works
FastTree: Computing Large Minimum Evolution Trees with Profiles instead of a Distance Matrix
Price, M. N., Dehal, P. S., & Arkin, A. P. (2009) · Molecular Biology and Evolution · doi:10.1093/molbev/msp077 · Cited by 5,602 · see citing works
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection
Kroenke, K., Spitzer, R. L., Williams, J. B. W., Monahan, P. O., & Löwe, B. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-146-5-200703060-00004 · Cited by 4,741 · see citing works
Receiver Operating Characteristic Curve in Diagnostic Test Assessment
Mandrekar, J. N. (2010) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e3181ec173d · Cited by 4,506 · see citing works
An Ultra-Brief Screening Scale for Anxiety and Depression: The PHQ–4
Kroenke, K., Spitzer, R. L., Williams, J. B. W., & Löwe, B. (2009) · Psychosomatics · doi:10.1016/s0033-3182(09)70864-3 · Cited by 4,261 · see citing works
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · see citing works
REVEL: An Ensemble Method for Predicting the Pathogenicity of Rare Missense Variants
Ioannidis, N. M., Rothstein, J. H., Pejaver, V., Middha, S., McDonnell, S. K., Baheti, S., Musolf, A., Li, Q., Holzinger, E., Karyadi, D., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.08.016 · Cited by 2,908 · see citing works
Yusuf, S., Hawken, S., Ôunpuu, S., Bautista, L., Franzosi, M. G., Commerford, P., Lang, C. C., Rumboldt, Z., Onen, C. L., Lisheng, L., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)67663-5 · Cited by 2,903 · see citing works
Glypican-1 identifies cancer exosomes and detects early pancreatic cancer
Melo, S. A., Luecke, L. B., Kahlert, C., Fernandez, A. F., Gammon, S. T., Kaye, J., LeBleu, V. S., Mittendorf, E. A., Weitz, J., Rahbari, N., et al. (2015) · Nature · doi:10.1038/nature14581 · Cited by 2,880 · see citing works
Time‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker
Heagerty, P. J., Lumley, T., & PEPE, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00337.x · Cited by 2,870 · see citing works
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · see citing works
Frailty as a predictor of surgical outcomes in older patients
Makary, M. A., Segev, D. L., Pronovost, P. J., Syin, D., Bandeen-Roche, K., Patel, P., Takenaga, R., Devgan, L., Holzmueller, C. G., Tian, J., et al. (2010) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2010.01.028 · Cited by 2,184 · see citing works
Logistic regression and Artificial Neural Network Classification models: A methodology review
Dreiseitl, S., & Ohno-Machado, L. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00034-0 · Cited by 2,177 · see citing works
Lauretani, F., Russo, C. R., Bandinelli, S., Bartali, B., Cavazzini, C., Di Iorio, A., Corsi, A. M., Rantanen, T., Guralnik, J. M., & Ferrucci, L. (2003) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00246.2003 · Cited by 1,946 · see citing works
Diagnostic Performance of Digital versus Film Mammography for Breast-Cancer Screening
Pisano, E. D., Gatsonis, C. A., Hendrick, E. R., Yaffe, M. J., Baum, J. K., Acharyya, S., Conant, E. F., Fajardo, L. L., Bassett, L. W., D'Orsi, C. J., et al. (2005) · New England Journal of Medicine · doi:10.1056/nejmoa052911 · Cited by 1,937 · see citing works
Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927 · see citing works
Arterial Stiffness and Risk of Coronary Heart Disease and Stroke: The Rotterdam Study
Mattace-Raso, F. U. S., van der Cammen, T. J. M., Hofman, A., van Popele, N. M., Bos, M. L., Schalekamp, M. A. D. H., Asmar, R., Reneman, R. S., Hoeks, A. P. G., Breteler, M. M. B., et al. (2006) · Circulation · doi:10.1161/circulationaha.105.555235 · Cited by 1,914 · see citing works
Simplified criteria for the diagnosis of autoimmune hepatitis
Hennes, E. M., Zeniya, M., Czaja, A. J., Parés, A., Dalekos, G. N., Krawitt, E. L., Bittencourt, P. L., Porta, G., Boberg, K. M., Hofer, H., et al. (2008) · Hepatology · doi:10.1002/hep.22322 · Cited by 1,901 · see citing works
Blanche, P., Dartigues, J., & Jacqmin‐Gadda, H. (2013) · Statistics in Medicine · doi:10.1002/sim.5958 · Cited by 1,877 · see citing works
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775 · see citing works
Evaluating presence–absence models in ecology: the need to account for prevalence
Manel, S., Williams, H. C., & Ormerod, S. J. (2001) · Journal of Applied Ecology · doi:10.1046/j.1365-2664.2001.00647.x · Cited by 1,759 · see citing works
Clinical impact of COVID-19 on patients with cancer (CCC19): a cohort study
Kuderer, N. M., Choueiri, T. K., Shah, D. P., Shyr, Y., Rubinstein, S. M., Rivera, D. R., Shete, S., Hsu, C.-Y., Desai, A., de Lima Lopes, G., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)31187-9 · Cited by 1,726 · see citing works
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
Rethinking receiver operating characteristic analysis applications in ecological niche modeling
Peterson, A. T., Papeş, M., & Soberón, J. M. (2008) · Ecological Modelling · doi:10.1016/j.ecolmodel.2007.11.008 · Cited by 1,715 · see citing works
High performance plasma amyloid-β biomarkers for Alzheimer's disease
Nakamura, A., Kaneko, N., Villemagne, V. L., Kato, T., Doecke, J., Doré, V., Fowler, C., Li, Q.-X., Martins, R., Rowe, C., et al. (2018) · Nature · doi:10.1038/nature25456 · Cited by 1,681 · see citing works
Early Diagnosis of Myocardial Infarction with Sensitive Cardiac Troponin Assays
Reichlin, T., Hochholzer, W., Bassetti, S., Steuer, S., Stelzig, C., Hartwiger, S., Biedert, S., Schaub, N., Buerge, C., Potocki, M., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0900428 · Cited by 1,678 · see citing works
Serdar, C. C., Cihan, M., Yücel, D., & Serdar, M. A. (2020) · Biochemia Medica · doi:10.11613/bm.2021.010502 · Cited by 1,678 · see citing works
EXPERT SYSTEMS WITH APPLICATIONS
Communication, S., Been-chian Chien, A., Jung Yi Lin, A., Tzung-pei Hong, B., & Com/locate/eswa, W. E. (2004) · Expert Systems with Applications · doi:10.1016/j.eswa.2004.10.010 · Cited by 1,670 · see citing works
Low Diagnostic Yield of Elective Coronary Angiography
Patel, M. R., Peterson, E. D., Dai, D., Brennan, J. M., Redberg, R. F., Anderson, H. V., Brindis, R. G., & Douglas, P. S. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa0907272 · Cited by 1,666 · see citing works
AUDIT‐C as a Brief Screen for Alcohol Misuse in Primary Care
Bradley, K. A., Rubinsky, A. D., Volk, R. J., Williams, E. C., Frank, D., & Kivlahan, D. R. (2007) · Alcoholism Clinical and Experimental Research · doi:10.1111/j.1530-0277.2007.00403.x · Cited by 1,634 · see citing works
Palmqvist, S., Janelidze, S., Quiroz, Y. T., Zetterberg, H., Lopera, F., Stomrud, E., Su, Y., Chen, Y., Serrano, G. E., Leuzy, A., et al. (2020) · JAMA · doi:10.1001/jama.2020.12134 · Cited by 1,603 · see citing works
Receiver operating characteristic curve: overview and practical use for clinicians
Nahm, F. S. (2022) · Korean Journal of Anesthesiology · doi:10.4097/kja.21209 · Cited by 1,569 · see citing works
Berg, W. A. (2008) · JAMA · doi:10.1001/jama.299.18.2151 · Cited by 1,527 · see citing works
Nørgaard, B. L., Leipsic, J. A., Gaur, S., SENEVIRATNE, S. K., Ko, B. S. H., Ito, H., Jensen, J. M., Mauri, L., Bruyne, B. D., Bezerra, H. G., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.043 · Cited by 1,511 · see citing works
Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Zou, K. H., O’Malley, A. J., & Mauri, L. (2007) · Circulation · doi:10.1161/circulationaha.105.594929 · Cited by 1,473 · see citing works
Severe hypoglycemia and risks of vascular events and death
Zoungas, S., Patel, A., Chalmers, J., de Galan, B. E., Li, Q., Billot, L., Woodward, M., Ninomiya, T., Neal, B., MacMahon, S., et al. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa1003795 · Cited by 1,471 · see citing works
Validation of Screening Questions for Limited Health Literacy in a Large VA Outpatient Population
Chew, L. D., Griffin, J. M., Partin, M. R., Noorbaloochi, S., Grill, J. P., Snyder, A., Bradley, K. A., Nugent, S. M., Simon, A. B., & van Ryn, M. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0520-5 · Cited by 1,432 · see citing works
Evaluation metrics and statistical tests for machine learning
Rainio, O., Teuho, J., & Klén, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-56706-x · Cited by 1,338 · see citing works
Pulmonary Hypertension in Heart Failure With Preserved Ejection Fraction
Lam, C. S. P., Roger, V. L., Rodeheffer, R. J., Borlaug, B. A., Enders, F., & Redfield, M. M. (2009) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2008.11.051 · Cited by 1,334 · see citing works
Musso, G., Gambino, R., Cassader, M., & Pagano, G. (2010) · Annals of Medicine · doi:10.3109/07853890.2010.518623 · Cited by 1,285 · see citing works
Pepe, M. S. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh101 · Cited by 1,276 · see citing works
The Diagnosis of Bronchopulmonary Dysplasia in Very Preterm Infants. An Evidence-based Approach
Jensen, E. A., Dysart, K., Gantz, M. G., McDonald, S. A., Bamat, N. A., Keszler, M., Kirpalani, H. M., Laughon, M. M., Poindexter, B. B., Duncan, A. F., et al. (2019) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201812-2348oc · Cited by 1,274 · see citing works
Iglesias, J. E., Augustinack, J. C., Nguyen, K., Player, C. M., Player, A., Wright, M., Roy, N., Frosch, M. P., McKee, A. C., Wald, L. L., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.04.042 · Cited by 1,267 · see citing works
Bruderer, R., Bernhardt, O. M., Gandhi, T., Miladinović, S. M., Cheng, L.-Y., Messner, S., Ehrenberger, T., Zanotelli, V., Butscheid, Y., Escher, C., et al. (2015) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m114.044305 · Cited by 1,238 · see citing works
Biomarkers of Cardiovascular Disease: Molecular Basis and Practical Considerations
Vasan, R. S. (2006) · Circulation · doi:10.1161/circulationaha.104.482570 · Cited by 1,212 · see citing works
Fisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165 · see citing works
Tsang, T. S. M., Barnes, M. E., Gersh, B. J., Bailey, K. R., & Seward, J. B. (2002) · The American Journal of Cardiology · doi:10.1016/s0002-9149(02)02864-3 · Cited by 1,159 · see citing works
Diagnostic Accuracy of Fractional Flow Reserve From Anatomic CT Angiography
Min, J. K., Leipsic, J. A., Pencina, M. J., Berman, D. S., Koo, B., van Mieghem, C. A. G., Ērglis, A., Lin, F. Y.-H., Dunning, A. M., Apruzzese, P. K., et al. (2012) · JAMA · doi:10.1001/2012.jama.11274 · Cited by 1,157 · see citing works
Yeboah, J., McClelland, R. L., Polonsky, T. S., Burke, G. L., Sibley, C. T., O’Leary, D., Carr, J. J., Goff, D. C., Greenland, P., & Herrington, D. M. (2012) · JAMA · doi:10.1001/jama.2012.9624 · Cited by 1,122 · see citing works
Altman, E. I., Iwanicz‐Drozdowska, M., Laitinen, E. K., & Suvas, A. (2016) · Journal of International Financial Management and Accounting · doi:10.1111/jifm.12053 · Cited by 1,111 · see citing works
Webster, L. R., & Webster, R. M. (2005) · Pain Medicine · doi:10.1111/j.1526-4637.2005.00072.x · Cited by 1,096 · see citing works
Artificial intelligence–enabled rapid diagnosis of patients with COVID-19
Mei, X., Lee, H.-C., Diao, K.-Y., Huang, M., Lin, B., Liu, C., Xie, Z., Ma, Y., Robson, P. M., Chung, M., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0931-3 · Cited by 1,084 · see citing works
LOWE, B. (2004) · Journal of Affective Disorders · doi:10.1016/s0165-0327(02)00237-9 · Cited by 1,060 · see citing works
Diagnosing schizophrenia in the initial prodromal phase
Klosterkötter, J., Hellmich, M., Steinmeyer, E. M., & Schultze-Lutter, F. (2001) · Archives of General Psychiatry · doi:10.1001/archpsyc.58.2.158 · Cited by 1,049 · see citing works
Sample size estimation in diagnostic test studies of biomedical informatics
Hajian‐Tilaki, K. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.013 · Cited by 1,012 · see citing works
Zhang, J., Gajjala, S., Agrawal, P., Tison, G. H., Hallock, L. A., Beussink-Nelson, L., Lassen, M. H., Fan, E., Aras, M. A., Jordan, C., et al. (2018) · Circulation · doi:10.1161/circulationaha.118.034338 · Cited by 1,006 · see citing works
“Left atrial volume, mL 4800 53 (40–71) 9 (16) 16 (29) 33 (66)”
Leach, M. O., Boggis, C. R. M., Dixon, A. K., Easton, D. F., Eeles, R. A., Evans, D. G., Gilbert, F. J., Griebsch, I., Hoff, R., Kessar, P., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)66481-1 · Cited by 1,005 · see citing works
Floegel, A., Stefan, N., Yu, Z., Mühlenbruch, K., Drogan, D., Joost, H.-G., Fritsche, A., Häring, H.-U., Hrabě de Angelis, M., Peters, A., et al. (2013) · Diabetes · doi:10.2337/db12-0495 · Cited by 988 · see citing works
Transferability and model evaluation in ecological niche modeling: a comparison of GARP and Maxent
Peterson, A. T., Papeş, M., & Eaton, M. D. (2007) · Ecography · doi:10.1111/j.0906-7590.2007.05102.x · Cited by 966 · see citing works
Pickhardt, P. J., Pooler, B. D., Lauder, T., del Rio, A. M., Bruce, R. J., & Binkley, N. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-8-201304160-00003 · Cited by 954 · see citing works
Ma, X.-J., Wang, Z., Ryan, P. D., Isakoff, S. J., Barmettler, A., Fuller, A. P., Muir, B., Mohapatra, G., Salunga, R. C., Tuggle, J. T., et al. (2004) · Cancer Cell · doi:10.1016/j.ccr.2004.05.015 · Cited by 945 · see citing works
The early prediction of mortality in acute pancreatitis: a large population-based study
Wu, B. U., Johannes, R. S., Sun, X., Tabak, Y. P., Conwell, D. L., & Banks, P. A. (2008) · Gut · doi:10.1136/gut.2008.152702 · Cited by 920 · see citing works
On the Rise of FinTechs: Credit Scoring Using Digital Footprints
Berg, T., Burg, V., Gombović, A., & Puri, M. (2020) · The Review of Financial Studies · doi:10.1093/rfs/hhz099 · Cited by 919 · see citing works
Receiver Operating Characteristic (ROC) Curve: Practical Review for Radiologists
Park, S. H., Goo, J. M., & Jo, C.-H. (2004) · Korean Journal of Radiology · doi:10.3348/kjr.2004.5.1.11 · Cited by 909 · see citing works
Benchmarking state-of-the-art classification algorithms for credit scoring
Baesens, B., Van Gestel, T., Viaene, S., Stepanova, M., Suykens, J., & Vanthienen, J. (2003) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601545 · Cited by 904 · see citing works
Genic Intolerance to Functional Variation and the Interpretation of Personal Genomes
Petrovski, S., Wang, Q., Heinzen, E. L., Allen, A. S., & Goldstein, D. B. (2013) · PLoS Genetics · doi:10.1371/journal.pgen.1003709 · Cited by 901 · see citing works
Zemek, R., Barrowman, N., Freedman, S. B., Gravel, J., Gagnon, I., McGahern, C., Aglipay, M., Sangha, G., Boutis, K., Beer, D., et al. (2016) · JAMA · doi:10.1001/jama.2016.1203 · Cited by 900 · see citing works
Brachial Flow-Mediated Dilation Predicts Incident Cardiovascular Events in Older Adults
Yeboah, J., Crouse, J. R., Hsu, F., Burke, G. L., & Herrington, D. M. (2007) · Circulation · doi:10.1161/circulationaha.106.678276 · Cited by 890 · see citing works
Davis, P. H., Dawson, J. D., Riley, W. A., & Lauer, R. M. (2001) · Circulation · doi:10.1161/hc4601.099486 · Cited by 881 · see citing works
Orlando, J. I., Fu, H., Barbosa‐Breda, J., Keer, K. V., Bathula, D. R., Diaz‐Pinto, A., Fang, R., Heng, P., Kim, J., Lee, J., et al. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101570 · Cited by 881 · see citing works
B-Type Natriuretic Peptide and Clinical Judgment in Emergency Diagnosis of Heart Failure
McCullough, P. A., Nowak, R. M., McCord, J., Hollander, J. E., Herrmann, H. C., Steg, P. G., Duc, P., Westheim, A. S., Omland, T., Knudsen, C. W., et al. (2002) · Circulation · doi:10.1161/01.cir.0000025242.79963.4c · Cited by 875 · see citing works
Receiver operating characteristic (ROC) curve for medical researchers
Kumar, R., & Indrayan, A. (2011) · Indian Pediatrics · doi:10.1007/s13312-011-0055-4 · Cited by 872 · see citing works
McQueen, M. J., Hawken, S., Wang, X., Ounpuu, S., Sniderman, A., Probstfield, J., Steyn, K., Sanderson, J. E., Hasani, M., Volkova, E., et al. (2008) · The Lancet · doi:10.1016/s0140-6736(08)61076-4 · Cited by 864 · see citing works
DISOPRED3: precise disordered region predictions with annotated protein-binding activity
Jones, D. T., & Cozzetto, D. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu744 · Cited by 861 · see citing works
Mason, S. J., & Graham, N. E. (2002) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/003590002320603584 · Cited by 860 · see citing works
Receiver Operating Characteristic Curves and Their Use in Radiology
Obuchowski, N. A. (2003) · Radiology · doi:10.1148/radiol.2291010898 · Cited by 843 · see citing works
The use of receiver operating characteristic curves in biomedical informatics
Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Lucila Ohno-machado, A. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.02.008 · Cited by 821 · see citing works
Distinguishing features of long COVID identified through immune profiling
Klein, J., Wood, J., Jaycox, J. R., Dhodapkar, R. M., Lu, P., Gehlhausen, J. R., Tabachnikova, A., Greene, K., TABACOF, L. R., Malik, A. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06651-y · Cited by 818 · see citing works
Cell-free DNA Analysis for Noninvasive Examination of Trisomy
Norton, M. E., Jacobsson, B., Swamy, G. K., Laurent, L. C., Ranzini, A. C., Brar, H., Tomlinson, M. W., Pereira, L., Spitz, J. L., Hollemon, D., et al. (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1407349 · Cited by 817 · see citing works
Imajo, K., Kessoku, T., Honda, Y., Tomeno, W., Ogawa, Y., Mawatari, H., Fujita, K., Yoneda, M., Taguri, M., Hyogo, H., et al. (2015) · Gastroenterology · doi:10.1053/j.gastro.2015.11.048 · Cited by 815 · see citing works
An Index Combining Respiratory Rate and Oxygenation to Predict Outcome of Nasal High-Flow Therapy
Roca, O., Caralt, B., Messika, J., Samper, M. A., Sztrymf, B., Hernández, G., García-de-Acilu, M., Frat, J., Masclans, J. R., & Ricard, J.-D. (2018) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201803-0589oc · Cited by 813 · see citing works
The Metabolic Syndrome as Predictor of Type 2 Diabetes
Lorenzo, C., Okoloise, M., Williams, K., Stern, M. P., & Haffner, S. M. (2003) · Diabetes Care · doi:10.2337/diacare.26.11.3153 · Cited by 800 · see citing works
Erbel, R. M., Möhlenkamp, S., Moebus, S., Schmermund, A., Lehmann, N., Stang, A., Dragano, N., Grönemeyer, D. H. W., Seibel, R. M. M., Kälsch, H., et al. (2010) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2010.06.030 · Cited by 775 · see citing works
Parental History of Myopia, Sports and Outdoor Activities, and Future Myopia
Jones, L. A., Sinnott, L. T., Mutti, D. O., Mitchell, G. L., Moeschberger, M. L., & Zadnik, K. (2007) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.06-1118 · Cited by 769 · see citing works
An experimental comparison of classification algorithms for imbalanced credit scoring data sets
Brown, I., & Mues, C. (2011) · Expert Systems with Applications · doi:10.1016/j.eswa.2011.09.033 · Cited by 768 · see citing works
Postoperative Biomarkers Predict Acute Kidney Injury and Poor Outcomes after Adult Cardiac Surgery
Parikh, C. R., Coca, S. G., Thiessen‐Philbrook, H., Shlipak, M. G., Koyner, J. L., Wang, Z., Edelstein, C. L., Devarajan, P., Patel, U. D., Zappitelli, M., et al. (2011) · Journal of the American Society of Nephrology · doi:10.1681/asn.2010121302 · Cited by 765 · see citing works
Genotype score in addition to common risk factors for prediction of type 2 diabetes
Meigs, J. B., Shrader, P., Sullivan, L. M., McAteer, J. B., Fox, C. S., Dupuis, J., Manning, A. K., Florez, J. C., Wilson, P. W. F., D'Agostino, R. B., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0804742 · Cited by 759 · see citing works
A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction
Yala, A., Lehman, C., Schuster, T., Portnoi, T., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2019182716 · Cited by 758 · see citing works
Stern, M. P., Williams, K., González-Villalpando, C., Hunt, K. J., & Haffner, S. M. (2004) · Diabetes Care · doi:10.2337/diacare.27.11.2676 · Cited by 735 · see citing works
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734 · see citing works
Cardiovascular Mortality Risk in Chronic Kidney Disease
Shlipak, M. G., Fried, L. F., Cushman, M., Manolio, T. A., Peterson, D., Stehman‐Breen, C. O., Bleyer, A. J., Newman, A. B., Siscovick, D. S., & Psaty, B. M. (2005) · JAMA · doi:10.1001/jama.293.14.1737 · Cited by 728 · see citing works
- Authors
- DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L.
- Affiliations
- IQVIA (United States) (US)
- IQVIA (United Kingdom) (GB)
- ORCID
- DeLong, Elizabeth R.
- Venue
- Biometrics
- Publisher
- JSTOR
- Published
- 1988-09
- Volume
- 44
- Issue
- 3
- Pages
- 837
- ISSN
- 0006-341X (ISSN-L)
- OpenAlex
- W2328176404
- Cited by
- 22,649
- References
- 12
- Citations by year
- 2026: 1897 · 2025: 1546 · 2024: 1644 · 2023: 1628 · 2022: 1861 · 2021: 1946 · 2020: 1651 · 2019: 1364 · 2018: 1193 · 2017: 1076 · 2016: 1001 · 2015: 924 · 2014: 775 · 2013: 745 · 2012: 621
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Biometrics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Health Services Research and Development
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- receiver operating characteristic, area under the ROC curve, diagnostic test comparison, nonparametric covariance estimation, sensitivity and false positive rate
- MeSH terms
- Algorithms, Analysis of Variance, Female, Humans, Intestinal Obstruction, Ovarian Neoplasms, Predictive Value of Tests, ROC Curve, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review