Cited by 22,649Imbalanced Data Classification TechniquesStatistical Methods in EpidemiologyStatistical Methods in Clinical Trials

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach

DeLong, Elizabeth R., DeLong, David M., Clarke-Pearson, Daniel L.·1988·Biometrics

DOI 10.2307/2531595

Open this paper in CoChat — free account

Abstract

Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.

Integrity signals

Registry

Crossref · 10.2307/2531595

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Basic principles of ROC analysis
Metz, C. E. (1978) · Seminars in Nuclear Medicine · doi:10.1016/s0001-2998(78)80014-2 · Cited by 6,122
A Class of Statistics with Asymptotically Normal Distribution
Hoeffding, W. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-0919-5_20 · Cited by 1,898
The area above the ordinal dominance graph and the area below the receiver operating characteristic graph
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,452
Untitled work
(1982) · Cited by 1,288
Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data
Dorfman, D. D., & Alf, E. F. (1969) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(69)90019-4 · Cited by 878
A New Approach for Testing the Significance of Differences Between ROC Curves Measured from Correlated Data
Metz, C. E., Wang, P.-L., & Kronman, H. B. (1984) · Information Processing in Medical Imaging · doi:10.1007/978-94-009-6045-9_25 · Cited by 359
Jackknifing $U$-Statistics
Arvesen, J. N. (1969) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697287 · Cited by 286
On Some Convergence Properties of UStatistics
Sen, P. K. (1960) · Calcutta Statistical Association Bulletin · doi:10.1177/0008068319600101 · Cited by 182
Surgical management of bowel obstruction in advanced ovarian carcinoma.
Krebs, H. B., & Goplerud, D. R. (1983) · PubMed · Cited by 132

Flagged first, then by citations

Show all 12 references

Sorted by citations

Items 1,101–1,197 of 1,197

Comparison of anthropometric and body composition measures as predictors of components of the metabolic syndrome in a clinical setting.
Mooney, S. J., Baecker, A., & Rundle, A. G. (2013) · Obesity Research & Clinical Practice · doi:10.1016/j.orcp.2012.10.004 · Cited by 54
Anti-Müllerian hormone in obese adolescent girls with polycystic ovary syndrome
Kim, J. Y., Tfayli, H., Michaliszyn, S. F., Lee, S., Nasr, A., & Arslanian, S. (2017) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2016.10.015 · Cited by 54
Update of the German Diabetes Risk Score and external validation in the German MONICA/KORA study
Mühlenbruch, K., Ludwig, T., Jeppesen, C., Joost, H.-G., Rathmann, W., Meisinger, C., Peters, A., Boeing, H., Thorand, B., & Schulze, M. B. (2014) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2014.03.013 · Cited by 52
Predicting tax avoidance by means of social network analytics
Lismont, J., Cardinaels, E., Bruynseels, L., De Groote, S., Baesens, B., Lemahieu, W., & Vanthienen, J. (2018) · Decision Support Systems · doi:10.1016/j.dss.2018.02.001 · Cited by 51
Comparative responsiveness of pain measures in cancer patients
Kroenke, K., Theobald, D., Wu, J., Tu, W., & Krebs, E. E. (2012) · The Journal of Pain · doi:10.1016/j.jpain.2012.05.004 · Cited by 49
Validation of the new 2021 EASL algorithm for the noninvasive diagnosis of advanced fibrosis in NAFLD
Canivet, C. M., Costentin, C., Irvine, K. M., Delamarre, A., Lannes, A., Sturm, N., Oberti, F., Patel, P. J., Decaens, T., Irles‐Depé, M., et al. (2023) · Hepatology · doi:10.1002/hep.32665 · Cited by 49
Developing and validating a prediction model for lymphedema detection in breast cancer survivors
Wei, X., Lu, Q., Jin, S., Li, F., Zhao, Q., Cui, Y., Jin, S., Cao, Y., & Fu, M. R. (2021) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2021.102023 · Cited by 49
Pre-Pregnancy Obesity vs. Other Risk Factors in Probability Models of Preeclampsia and Gestational Hypertension
Lewandowska, M., Więckowska, B., Sajdak, S., & Lubiński, J. (2020) · Nutrients · doi:10.3390/nu12092681 · Cited by 49
Symptoms do not correlate with findings from colonoscopy in children with inflammatory bowel disease and primary sclerosing cholangitis
Ricciuto, A., Fish, J., Carman, N., Walters, T. D., Church, P. C., Hansen, B. E., Crowley, E., Siddiqui, I., Nguyen, G. C., Kamath, B. M., et al. (2018) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2018.01.020 · Cited by 47
The primary health care version of ICD-11: the detection of common mental disorders in general medical settings
Goldberg, D. P., Prisciandaro, J. J., & Williams, P. (2012) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2012.06.006 · Cited by 48
Artificial Intelligence-Enabled End-To-End Detection and Assessment of Alzheimer’s Disease Using Voice
Agbavor, F., & Liang, H. (2022) · Brain Sciences · doi:10.3390/brainsci13010028 · Cited by 47
Plasma glycated CD59, a novel biomarker for detection of pregnancy-induced glucose intolerance
Ghosh, P., Luque-Fernandez, M. A., Vaidya, A., Ma, D., Sahoo, R., Chorev, M., Zera, C., McElrath, T. F., Williams, M. A., Seely, E. W., et al. (2017) · Diabetes Care · doi:10.2337/dc16-2598 · Cited by 48
“(20) nonparametric tied corrected estimator, and the percentile values of the case patient observations with respect to the control distributionwere used to derive the tied corrected estimator (20).”
Method
Explainable machine learning aggregates polygenic risk scores and electronic health records for Alzheimer’s disease prediction
Gao, X. R., Chiariglione, M., Qin, K., Nuytemans, K., Scharre, D. W., Li, Y.-J., & Martin, E. R. (2023) · Scientific Reports · doi:10.1038/s41598-023-27551-1 · Cited by 45
Evaluation of the American Heart Association cardiovascular disease prevention guideline for women
Hsia, J., Rodabough, R. J., Manson, J. E., Liu, S., Freiberg, M. S., Graettinger, W., Rosal, M. C., Cochrane, B., Lloyd-Jones, D., Robinson, J. G., et al. (2010) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.108.842385 · Cited by 45
Performance Characteristics of Combinations of Host Biomarkers to Identify Women with Occult Placental Malaria: A Case-Control Study from Malawi
Conroy, A. L., Liles, W. C., Molyneux, M. E., Rogerson, S. J., & Kain, K. C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028540 · Cited by 46
Effect of obesity and morbid obesity on the growth hormone (GH) secretion elicited by the combined GHRH + GHRP-6 test
Kelestimur, F., Popovic, V., Leal, A., Van Dam, P. S., Torres, E., Perez Mendez, L. F., Greenman, Y., Koppeschaar, H. P. F., Dieguez, C., & Casanueva, F. F. (2006) · Clinical Endocrinology · doi:10.1111/j.1365-2265.2006.02525.x · Cited by 45
Validation of Surrogate Anthropometric Indices in Older Adults: What Is the Best Indicator of High Cardiometabolic Risk Factor Clustering?
Ramírez-Vélez, R., Pérez-Sousa, M. Á., Izquierdo, M., Cano-Gutierrez, C. A., González-Jiménez, E., Schmidt-RioValle, J., González-Ruíz, K., & Correa-Rodríguez, M. (2019) · Nutrients · doi:10.3390/nu11081701 · Cited by 45
Accuracy of Serial National Institutes of Health Stroke Scale Scores to Identify Artery Status in Acute Ischemic Stroke
Mikulik, R., Ribo, M., Hill, M. D., Grotta, J. C., Malkoff, M., Molina, C., Rubiera, M., Delgado-Mederos, R., Alvarez-Sabin, J., & Alexandrov, A. V. (2007) · Circulation · doi:10.1161/circulationaha.106.651026 · Cited by 43
One step further into the blackbox: A pilot study of how to build more confidence around an AI-based decision system of breast nodule assessment in 2D ultrasound
Dong, F., She, R., Cui, C., Shi, S., Hu, X., Zeng, J., Wu, H., Xu, J., & Zhang, Y. (2021) · European Radiology · doi:10.1007/s00330-020-07561-7 · Cited by 43
Prognostic Value of Circulating MicroRNA-210 Levels in Patients with Moderate to Severe Aortic Stenosis
Røsjø, H., Dahl, M. B., Bye, A., Andreassen, J., Jørgensen, M., Wisløff, U., Christensen, G., Edvardsen, T., & Omland, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0091812 · Cited by 41
Domain Shifts in Machine Learning Based Covid-19 Diagnosis From Blood Tests
Roland, T., Böck, C., Tschoellitsch, T., Maletzky, A., Hochreiter, S., Meier, J., & Klambauer, G. (2022) · Journal of Medical Systems · doi:10.1007/s10916-022-01807-1 · Cited by 42
The Role of GDF-15 in Heart Failure Patients With Chronic Kidney Disease
Benes, J., Kotrc, M., Wohlfahrt, P., Conrad, M. J., Franekova, J., Jabor, A., Lupinek, P., Kautzner, J., Melenovsky, V., & Jarolim, P. (2019) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2018.12.027 · Cited by 42
Cerebrospinal fluid α‐synuclein contributes to the differential diagnosis of Alzheimer's disease
Shi, M., Tang, L., Toledo, J. B., Ginghina, C., Wang, H., Aro, P., Jensen, P. H., Weintraub, D., Chen‐Plotkin, A. S., Irwin, D. J., et al. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2018.02.015 · Cited by 42
Detecting glaucoma based on spectral domain optical coherence tomography imaging of peripapillary retinal nerve fiber layer: a comparison study between hand-crafted features and deep learning model
Zheng, C., Xie, X., Huang, L., Chen, B., Yang, J., Lu, J., Qiao, T., Fan, Z., & Zhang, M. (2020) · Graefe's Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-019-04543-4 · Cited by 40
Identification of hypertrophic cardiomyopathy on electrocardiographic images with deep learning
Sangha, V., Dhingra, L. S., Aminorroaya, A., Croon, P. M., Sikand, N. V., Sen, S., Martinez, M. W., Maron, M. S., Krumholz, H. M., Asselbergs, F. W., et al. (2025) · Nature Cardiovascular Research · doi:10.1038/s44161-025-00685-3 · Cited by 41
Assessing Adolescent Anxiety in General Psychiatric Care: Diagnostic Accuracy of the Swedish Self-Report and Parent Versions of the Spence Children’s Anxiety Scale
Olofsdotter, S., Sonnby, K., Vadlin, S., Furmark, T., & Nilsson, K. W. (2016) · Assessment · doi:10.1177/1073191115583858 · Cited by 41
Follow-up of pediatric celiac disease: value of antibodies in predicting mucosal healing, a prospective cohort study
Vécsei, E., Steinwendner, S., Kogler, H., Innerhofer, A., Hammer, K., Haas, O. A., Amann, G., Chott, A., Vogelsang, H., Schoenlechner, R., et al. (2014) · BMC Gastroenterology · doi:10.1186/1471-230x-14-28 · Cited by 41
Correction: Systemic Complement Activation in Age-Related Macular Degeneration
Scholl, H. P. N., Charbel Issa, P., Walier, M., Janzer, S., Pollok-Kopp, B., Börncke, F., Fritsche, L. G., Chong, N. V., Fimmers, R., Wienker, T., et al. (2008) · PLoS ONE · doi:10.1371/annotation/32b9bc31-ed6d-4d31-9ce0-480407017bad · Cited by 39
Assessing the incremental role of novel and emerging risk factors.
Cook, N. R. (2010) · Current Cardiovascular Risk Reports · doi:10.1007/s12170-010-0084-x · Cited by 39
Intraplacental Fetal Vessel Diameter May Help Predict for Placental Invasiveness in Pregnant Women at High Risk for Placenta Accreta Spectrum Disorders
Bourgioti, C., Konstantinidou, A. E., Zafeiropoulou, K., Antoniou, A., Fotopoulos, S., Theodora, M., Daskalakis, G., Nikolaidou, M. E., Tzavara, C., Letsika, A., et al. (2021) · Radiology · doi:10.1148/radiol.2020200273 · Cited by 38
Incremental value of serum neurofilament light chain and glial fibrillary acidic protein as blood-based biomarkers for predicting functional outcome in severe acute ischemic stroke
Vollmuth, C., Fiessler, C., Montellano, F. A., Kollikowski, A. M., Essig, F., Oeckl, P., Barba, L., Steinacker, P., Schulz, C., Ungethüm, K., et al. (2024) · European Stroke Journal · doi:10.1177/23969873241234436 · Cited by 38
Preoperative data-based deep learning model for predicting postoperative survival in pancreatic cancer patients
Lee, W., Park, H. J., Lee, H.-J., Jun, E., Song, K. B., Hwang, D. W., Lee, J. H., Lim, K., Kim, N., Lee, S. S., et al. (2022) · International Journal of Surgery · doi:10.1016/j.ijsu.2022.106851 · Cited by 35
Non-calcific aortic tissue quantified from computed tomography angiography improves diagnosis and prognostication of patients referred for transcatheter aortic valve implantation
Grodecki, K., Tamarappoo, B. K., Huczek, Z., Jedrzejczyk, S., Cadet, S., Kwiecinski, J., Rymuza, B., Parma, R., Olasinska-Wisniewska, A., Fijalkowska, J., et al. (2021) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jeaa304 · Cited by 38
Serum 1,5-anhydroglucitol level as a screening tool for diabetes mellitus in a community-based population at high risk of diabetes
Wang, Y., Yuan, Y., Zhang, Y., Lei, C., Zhou, Y., He, J., & Sun, Z. (2017) · Acta Diabetologica · doi:10.1007/s00592-016-0944-z · Cited by 37
Novel Approach to Fecal Occult Blood Testing by Assay of Erythrocyte-Specific microRNA Markers
Wu, C. W., Cao, X., Berger, C. K., Foote, P. H., Mahoney, D. W., Simonson, J. A., Anderson, B. W., Yab, T. C., Taylor, W. R., Boardman, L. A., et al. (2017) · Digestive Diseases and Sciences · doi:10.1007/s10620-017-4627-6 · Cited by 37
Machine Learning Approach to Classify Cardiovascular Disease in Patients With Nonalcoholic Fatty Liver Disease in the UK Biobank Cohort
Sharma, D., Gotlieb, N., Farkouh, M. E., Patel, K., Xu, W., & Bhat, M. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.121.022576 · Cited by 36
A Comparison of Skin Endpoint Titration and Skin-Prick Testing in the Diagnosis of Allergic Rhinitis
Gungor, A., Houser, S. M., Aquino, B. F., Akbar, I., Moinuddin, R., Mamikoglu, B., & Corey, J. P. (2004) · Ear, Nose & Throat Journal · doi:10.1177/014556130408300118 · Cited by 36
Predicting the tumor microenvironment composition and immunotherapy response in non-small cell lung cancer from digital histopathology images
Patkar, S., Chen, A., Basnet, A., Bixby, A., Rajendran, R., Chernet, R., Faso, S., Kumar, P. A., Desai, D., El-Zammar, O., et al. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00765-w · Cited by 41
“95% Con ﬁ dence intervals for AUROC were computed by the DeLong approach 79 .”
Evaluation of Federated learning variations for COVID-19 diagnosis using chest Radiographs from 42 US and European hospitals
Peng, L., Luo, G., Walker, A., Zaiman, Z., Jones, E. K., Gupta, H., Kersten, K., Burns, J. L., Harle, C. A., Magoc, T., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac188 · Cited by 36
Levator hiatal area as a risk factor for cystocele recurrence after surgery: a prospective study
Vergeldt, T., Notten, K., Weemhoff, M., van Kuijk, S., Mulder, F., Beets‐Tan, R., Vliegen, R., Gondrie, E., Bergmans, M., Roovers, J., et al. (2015) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13340 · Cited by 35
Accurate and Feasible Deep Learning Based Semi-Automatic Segmentation in CT for Radiomics Analysis in Pancreatic Neuroendocrine Neoplasms
Huang, B., Lin, X., Shen, J., Chen, X., Chen, J., Li, Z.-P., Wang, M., Yuan, C., Diao, X.-F., Luo, Y., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3070708 · Cited by 35
Lower DNA methylation levels in CpG island shores of CR1, CLU, and PICALM in the blood of Japanese Alzheimer’s disease patients
Mitsumori, R., Sakaguchi, K., Shigemizu, D., Mori, T., Akiyama, S., Ozaki, K., Niida, S., & Shimoda, N. (2020) · PLOS ONE · doi:10.1371/journal.pone.0239196 · Cited by 34
External validation and extension of the NAPLS-2 and SIPS-RC personalized risk calculators in an independent clinical high-risk sample
Osborne, K. J., & Mittal, V. A. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.06.034 · Cited by 31
Can shape analysis differentiate free-floating internal carotid artery Thrombus from atherosclerotic plaque in patients evaluated with CTA?for stroke or transient ischemic attack?
Thornhill, R. E., Lum, C., Jaberi, A., Stefanski, P., Torres, C. H., Momoli, F., Petrcich, W., & Dowlatshahi, D. (2014) · Academic Radiology · doi:10.1016/j.acra.2013.11.011 · Cited by 33
AttriMIL: Revisiting attention-based multiple instance learning for whole-slide pathological image classification from a perspective of instance attributes
Cai, L., Huang, S., Zhang, Y., Lu, J., & Zhang, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103631 · Cited by 33
Toward AI-supported US Triage of Women with Palpable Breast Lumps in a Low-Resource Setting
Berg, W. A., López Aldrete, A.-L., Jairaj, A., Ledesma Parea, J. C., García, C. Y., McClennan, R. C., Cen, S. Y., Larsen, L. H., de Lara, M. T. S., & Love, S. (2023) · Radiology · doi:10.1148/radiol.223351 · Cited by 36
Implication of hypocholesterolemia in autism spectrum disorder and its associated comorbidities: A retrospective case–control study
Benachenhou, S., Etcheverry, A., Galarneau, L., Dubé, J., & Çaku, A. (2019) · Autism Research · doi:10.1002/aur.2183 · Cited by 33
Texture analysis of FDG PET/CT for differentiating between FDG-avid benign and metastatic adrenal tumors: efficacy of combining SUV and texture parameters
Nakajo, M., Jinguji, M., Nakajo, M., Shinaji, T., Nakabeppu, Y., Fukukura, Y., & Yoshiura, T. (2017) · Abdominal Radiology · doi:10.1007/s00261-017-1207-3 · Cited by 32
Clinical and biological risk factors for neuropsychological impairment in alcohol use disorder
Ritz, L., Coulbault, L., Lannuzel, C., Boudehent, C., Segobin, S., Eustache, F., Vabret, F., Pitel, A. L., & Beaunieux, H. (2016) · PLOS ONE · doi:10.1371/journal.pone.0159616 · Cited by 33
Deep learning in automatic detection of dysphonia: comparing acoustic features and developing a generalizable framework
Chen, Z., Zhu, P., Qiu, W., Guo, J., & Li, Y. (2023) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.12783 · Cited by 31
Prior CT Improves Deep Learning for Malignancy Risk Estimation of Screening-detected Pulmonary Nodules
Vaidhya Venkadesh, K., Aleef, T. A., Scholten, E. T., Saghir, Z., Silva, M., Sverzellati, N., Pastorino, U., van Ginneken, B., Prokop, M., & Jacobs, C. (2023) · Radiology · doi:10.1148/radiol.223308 · Cited by 32
Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions
Hinson, J. S., Klein, E., Smith, A., Toerper, M., Dungarani, T., Hager, D., Hill, P., Kelen, G., Niforatos, J. D., Stephens, R. S., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00646-1 · Cited by 30
Development and clinical evaluation of an artificial intelligence support tool for improving telemedicine photo quality
Vodrahalli, K., Ko, J., Chiou, A. S., Novoa, R., Abid, A., Phung, M., Yekrang, K., Petrone, P., Zou, J., & Daneshjou, R. (2023) · JAMA Dermatology · doi:10.1001/jamadermatol.2023.0091 · Cited by 30
A high sensitivity assay is more accurate than a classical assay for the measurement of plasma CRP levels in endometriosis
Vodolazkaia, A., Bossuyt, X., Fassbender, A., Kyama, C. M., Meuleman, C., Peeraer, K., Tomassetti, C., & D'Hooghe, T. M. (2011) · Reproductive Biology and Endocrinology · doi:10.1186/1477-7827-9-113 · Cited by 28
Perioperative risk prediction in the era of enhanced recovery: a comparison of POSSUM, ACPGBI, and E-PASS scoring systems in major surgical procedures of the colorectal surgeon.
Bagnall, N. M., Pring, E. T., Malietzis, G., Athanasiou, T., Faiz, O. D., Kennedy, R. H., & Jenkins, J. T. (2018) · International Journal of Colorectal Disease · doi:10.1007/s00384-018-3141-4 · Cited by 27
Aortic size predicts aortic dissection in Turner syndrome—a 25-year prospective cohort study
Thunström, S., Thunström, E., Naessén, S., Berntorp, K., Laczna Kitlinski, M., Ekman, B., Wahlberg, J., Bergström, I., Bech-Hanssen, O., Krantz, E., et al. (2023) · International Journal of Cardiology · doi:10.1016/j.ijcard.2022.11.023 · Cited by 26
In vivo validity of proximal caries detection in primary teeth, with histological validation
Subka, S., Rodd, H., Nugent, Z., & Deery, C. (2019) · International Journal of Paediatric Dentistry · doi:10.1111/ipd.12478 · Cited by 25
Associations Between Macrophage Colony‐Stimulating Factor and Monocyte Chemotactic Protein 1 in Plasma and First‐Time Coronary Events: A Nested Case–Control Study
Schiopu, A., Bengtsson, E., Gonçalves, I., Nilsson, J., Fredrikson, G. N., & Björkbacka, H. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.115.002851 · Cited by 25
Anatomical Cystocele Recurrence: Development and Internal Validation of a Prediction Model
Vergeldt, T. F. M., van Kuijk, S. M. J., Notten, K. J. B., Kluivers, K. B., & Weemhoff, M. (2016) · Obstetrics & Gynecology · doi:10.1097/aog.0000000000001272 · Cited by 25
The Development and Internal Evaluation of a Predictive Model to Identify for Whom Mindfulness-Based Cognitive Therapy Offers Superior Relapse Prevention for Recurrent Depression Versus Maintenance Antidepressant Medication
Cohen, Z. D., DeRubeis, R. J., Hayes, R., Watkins, E. R., Lewis, G., Byng, R., Byford, S., Crane, C., Kuyken, W., Dalgleish, T., et al. (2023) · Clinical Psychological Science · doi:10.1177/21677026221076832 · Cited by 25
A deep learning fusion network trained with clinical and high-frequency ultrasound images in the multi-classification of skin diseases in comparison with dermatologists: a prospective and multicenter study
Zhu, A.-Q., Wang, Q., Shi, Y.-L., Ren, W.-W., Cao, X., Ren, T.-T., Wang, J., Zhang, Y.-Q., Sun, Y.-K., Chen, X.-W., et al. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102391 · Cited by 25
Artificial Intelligence in Health Data Analysis: The Darwinian Evolution Theory Suggests an Extremely Simple and Zero-Cost Large-Scale Screening Tool For Prediabetes and Type 2 Diabetes
Buccheri, E., Dell'Aquila, D., & Russo, M. (2021) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2021.108722 · Cited by 24
Circulating sphingosine-1-phosphate as a prognostic biomarker for community-acquired pneumonia
Hsu, S.-C., Chang, J.-H., Hsu, Y.-P., Bai, K.-J., Huang, S.-K., & Hsu, C.-W. (2019) · PLOS ONE · doi:10.1371/journal.pone.0216963 · Cited by 23
TN‐USMA Net: Triple normalization‐based gastrointestinal stromal tumors classification on multicenter EUS images with ultrasound‐specific pretraining and meta attention
Liu, C., Qiao, M., Jiang, F., Guo, Y., Jin, Z., & Wang, Y. (2021) · Medical Physics · doi:10.1002/mp.15172 · Cited by 22
Integrating health data-driven machine learning algorithms to evaluate risk factors of early stage hypertension at different levels of HDL and LDL cholesterol
Liao, P.-C., Chen, M.-S., Jhou, M.-J., Chen, T.-C., Yang, C.-T., & Lu, C.-J. (2022) · Diagnostics · doi:10.3390/diagnostics12081965 · Cited by 21
A comprehensive WGS-based pipeline for the identification of new candidate genes in inherited retinal dystrophies
González-del Pozo, M., Fernández-Suárez, E., Bravo-Gil, N., Méndez-Vidal, C., Martín-Sánchez, M., Rodríguez-de la Rúa, E., Ramos-Jiménez, M., Morillo-Sánchez, M. J., Borrego, S., & Antiñolo, G. (2022) · npj Genomic Medicine · doi:10.1038/s41525-022-00286-0 · Cited by 21
Acute phase proteins, interleukin-6, tumor necrosis factor, nitric oxide and oxidative stress markers in horses with cutaneous habronemosis under field condition
El-Deeb, W., Iacob, O., Fayez, M., Elgioushy, M., Shawaf, T., & Ibrahim, A. (2018) · Veterinary Parasitology · doi:10.1016/j.vetpar.2018.03.023 · Cited by 17
Stacking machine learning algorithms for biomarker-based preoperative diagnosis of a pelvic mass
Shaw, R., Lokshin, A. E., Miller, M. C., Messerlian-Lambert, G., & Moore, R. G. (2022) · Cancers · doi:10.3390/cancers14051291 · Cited by 16
Validation of the NoSAS Score for the Screening of Sleep-Disordered Breathing in a Sleep Clinic
Rong, Y., Wang, S., Wang, H., Wang, F., Tang, J., Kang, X., Li, G., & Liu, Z. (2020) · Canadian Respiratory Journal · doi:10.1155/2020/4936423 · Cited by 15
Benchmarking artificial intelligence methods for end-to-end computational pathology
Laleh, N. G., Muti, H. S., Lavinia Loeffler, C. M., Echle, A., Saldanha, O. L., Mahmood, F., Lu, M. Y., Trautwein, C., Langer, R., Dislich, B., et al. (2021) · doi:10.1101/2021.08.09.455633 · Cited by 13
Detecting racial/ethnic health disparities using deep learning from frontal chest radiography
Pyrros, A., Rodríguez-Fernández, J. M., Borstelmann, S. M., Gichoya, J. W., Horowitz, J. M., Fornelli, B., Siddiqui, N., Velichko, Y., Koyejo, O., & Galanter, W. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.09.010 · Cited by 15
“…classi ﬁ er ’ s predictions for vascular disease compared with the EHR-based administrative data (HCC-108) showed areas under the curve of 0.851 (95% CI, 0.816-0.887) in the ambulatory cohort and 0.694 (95% CI, 0.641-0.748) in the hospitalized cohort, with P .001, using DeLong ’ s method [16].”
MethodInfluential
Deep learning improves utility of tau PET in the study of Alzheimer's disease
Zou, J., Park, D., Johnson, A., Feng, X., Pardo, M., France, J., Tomljanovic, Z., Brickman, A. M., Devanand, D. P., Luchsinger, J. A., et al. (2021) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1002/dad2.12264 · Cited by 15
Systematic investigation of machine learning on limited data: A study on predicting protein-protein binding strength
Zheng, F., Jiang, X., Wen, Y., Yang, Y., & Li, M. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.12.018 · Cited by 14
Dexamethasone suppression test predicts later development of an impaired adrenal function after a 14-day course of prednisone in healthy volunteers
Neidert, S., Schuetz, P., Mueller, B., & Christ-Crain, M. (2010) · European Journal of Endocrinology · doi:10.1530/eje-09-0930 · Cited by 13
True Chromogranin A concentrations in plasma from patients with small intestinal neuroendocrine tumours
Rehfeld, J. F., Broedbaek, K., Goetze, J. P., Knigge, U., & Hilsted, L. M. (2020) · Scandinavian Journal of Gastroenterology · doi:10.1080/00365521.2020.1759141 · Cited by 13
Holistic AI analysis of hybrid cardiac perfusion images for mortality prediction
Marcinkiewicz, A. M., Zhang, W., Shanbhag, A., Miller, R. J. H., Lemley, M., Ramirez, G., Buchwald, M., Killekar, A., Kavanagh, P. B., Feher, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01526-0 · Cited by 12
Deep Genomic Signature for early metastasis prediction in prostate cancer
Sharifi-Noghabi, H., Liu, Y., Erho, N., Shrestha, R., Alshalalfa, M., Davicioni, E., Collins, C. C., & Ester, M. (2018) · doi:10.1101/276055 · Cited by 11
B-Type Natriuretic Peptides and High-Sensitive Troponin I as COVID-19 Survival Factors: Which One Is the Best Performer?
de Falco, R., Vargas, M., Palma, D., Savoia, M., Miscioscia, A., Pinchera, B., Vano, M., Servillo, G., Gentile, I., & Fortunato, G. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10122726 · Cited by 11
PENSIEVE-AI a brief cognitive test to detect cognitive impairment across diverse literacy
Liew, T. M., Foo, J. Y. H., Yang, H., Tay, S. Y., Koay, W. I., Yip, K. F., Ting, S. K. S., Narasimhalu, K., Li, W., Tan, C., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58201-x · Cited by 11
Salivary cortisol and cortisone in diagnosis of Cushing's syndrome: a comparison of six different analytical methods
Bäcklund, N., Brattsand, G., Lundstedt, S., Aardal, E., Bartuseviciene, I., Berinder, K., Höybye, C., Burman, P., Edén Engström, B., Isaksson, A., et al. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2023-0141 · Cited by 9
Development and validation of a multimodal-based prognosis and intervention prediction model for COVID-19 patients in a multicenter cohort
Lee, J. H., Ahn, J. S., Chung, M. J., Jeong, Y. J., Kim, J. H., Lim, J. K., Kim, J. Y., Kim, Y. J., Lee, J. E., & Kim, E. Y. (2022) · Sensors · doi:10.3390/s22135007 · Cited by 9
Succoring the challenging acute mesenteric ischemia: Feasibility of lactate dehydrogenase for evaluation of intestinal necrosis extension and mortality
Monroy, D. M. C., Arango, F. G., Moreno, L. R., Chaves, C. E. R., Donoso-Samper, A., Nassar, R., & Isaza-Restrepo, A. (2022) · Annals of Medicine & Surgery · doi:10.1016/j.amsu.2022.104922 · Cited by 8
Consensus meta-analysis of genome-wide association studies for Alzheimer’s disease and related dementias
EADB, Bellenguez, C., Castillo Morales, A., Amin, N., van der Lee, S. J., Muntaner, M., Parveen, K., Küçükali, F., Grenier-Boley, B., Heikkinen, S., et al. (2026) · Nature Genetics · doi:10.1038/s41588-026-02583-1 · Cited by 7
Artificial Intelligence Electrocardiogram and Left Ventricular Systolic Dysfunction in Kenya
Pandey, A., Keshvani, N., Segar, M. W., Kwon, J.-M., Lee, H. S., Bhograj, C., Mashilane, K. I., Jain, N., Mwiti, W., Wambari, E., et al. (2026) · JAMA Cardiology · doi:10.1001/jamacardio.2026.0908 · Cited by 7
The Role of Combined Muscle Ultrasound and Bioimpedentiometry Parameters for Sarcopenia Diagnosis in a Population of Hospitalized Older Adults
Zanotelli, A., Rossi, A. P., Del Monte, L., Vantini, G., Stabile, G., Urbani, S., Giani, A., Zoico, E., Babbanini, A., Fantin, F., et al. (2024) · Nutrients · doi:10.3390/nu16152429 · Cited by 6
Validation of a Retrospective Computing Model for Mortality Risk in the Intensive Care Unit
Tan, E. M., Kashyap, R., Olson, I. C., & O’Horo, J. C. (2020) · Mayo Clinic Proceedings: Innovations, Quality & Outcomes · doi:10.1016/j.mayocpiqo.2020.09.001 · Cited by 5
Criterion Validity of the WAIS-IV Cognitive Proficiency Index (CPI)
Logue, E., Scarisbrick, D. M., Thaler, N. S., Mahoney, J. J., Block, C. K., Adams, R., & Scott, J. (2015) · The Clinical Neuropsychologist · doi:10.1080/13854046.2015.1101490 · Cited by 4
Capillary ketone level and future ketoacidosis risk in patients with type 1 diabetes using sodium-glucose cotransporter inhibitors
Bapat, P., Dhaliwal, S., Song, C., Zhang, Y., Scarr, D., Bakhsh, A., Budhram, D., Verhoeff, N. J., Weisman, A., Fralick, M., et al. (2025) · Diabetes Care · doi:10.2337/dc25-0125 · Cited by 4
Predicting Food Insecurity in a Pediatric Population Using the Electronic Health Record
Rigdon, J., Montez, K., Palakshappa, D., Brown, C., Downs, S. M., Albertini, L. W., & Taxter, A. (2024) · Journal of Clinical and Translational Science · doi:10.1017/cts.2024.645 · Cited by 3
Development and Validation of Computable Social Phenotypes for Health‐Related Social Needs
Gregory, M. E., Kasthurirathne, S. N., Magoc, T., McNamee, C., Harle, C. A., & Vest, J. R. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae150 · Cited by 2
Digital quantification of Ki67 and PRAME in challenging melanocytic lesions—A novel diagnostic tool
Brogård, M. B., Steiniche, T., Lade-Keller, J., Wandler, A., Christensen, K. B., Georgsen, J. B., & Nielsen, P. S. (2025) · Pathology - Research and Practice · doi:10.1016/j.prp.2025.155953 · Cited by 1
MuST: Multimodal Spatiotemporal Graph-Transformer for Hospital Readmission Prediction
Miao, Y., & Yu, L. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-47425-5_25 · Cited by 1
Linguistic traces of stochastic empathy in language models
Kleinberg, B., Zegers, J., Festor, J., Vida, S., Präsent, J., Loconte, R., & Peereboom, S. (2024) · arXiv · doi:10.48550/arxiv.2410.01675 · Cited by 1
“All data, materials and code to reproduce the LLM data collection are publicly available at: https://osf.io/rcz6s/?view_only=82e087f3c0a64b0aaf6039094cfc5c9c.”
AI-Driven Predictions of Readmission and Mortality for Improved Discharge Decisions in Critical Care: A Retrospective Study
Heo, Y., Kim, M., Han, S.-S., Kim, T.-H., Heo, J., Kim, D., Kim, W. J., Lee, S.-J., Kwon, O. B., Kim, Y., et al. (2026) · Diagnostics · doi:10.3390/diagnostics16060874
“We performed the DeLong test [23] to compare the statistical significance of AUROCs between SWIFT and GRU-D++.”
Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation
Gupta, S., & Singla, J. (2026) · arXiv · doi:10.48550/arxiv.2609.10495
“Pairwise comparisons between correlated ROC curves were performed using DeLong’s test [26] to determine whether observed differences between competing quality estimation methods were statistically significant.”
Plasma proteomics enhances genetic risk prediction and biological understanding of venous thromboembolism
Mei, Y., Vad, O. B., Beyer, A. F., Ghouse, J., & Lundegaard, P. R. (2026) · Molecular & Cellular Proteomics · doi:10.1016/j.mcpro.2026.101692
Untitled work
(2013) · Europe PMC (PubMed Central) · Cited by 2,089

Authors: DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L.
Affiliations: IQVIA (United States) (US)
IQVIA (United Kingdom) (GB)
ORCID: DeLong, Elizabeth R.
Venue: Biometrics
Publisher: JSTOR
Published: 1988-09
Volume: 44
Issue: 3
Pages: 837
ISSN: 0006-341X (ISSN-L)
DOI: 10.2307/2531595
OpenAlex: W2328176404
Cited by: 22,649
References: 12
Citations by year: 2026: 1897 · 2025: 1546 · 2024: 1644 · 2023: 1628 · 2022: 1861 · 2021: 1946 · 2020: 1651 · 2019: 1364 · 2018: 1193 · 2017: 1076 · 2016: 1001 · 2015: 924 · 2014: 775 · 2013: 745 · 2012: 621
FWCI: 4.52
Type: journal article
Language: en
Open access: closed
Locations: Biometrics (publishedVersion)
PubMed (publishedVersion)
Funders: Health Services Research and Development
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords: receiver operating characteristic, area under the ROC curve, diagnostic test comparison, nonparametric covariance estimation, sensitivity and false positive rate
MeSH terms: Algorithms, Analysis of Variance, Female, Humans, Intestinal Obstruction, Ovarian Neoplasms, Predictive Value of Tests, ROC Curve, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks