The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves
Abstract
Among algorithms for comparing the areas under two or more correlated receiver operating characteristic (ROC) curves, DeLong's algorithm is perhaps the most widely used one due to its simplicity of implementation in practice. Unfortunately, however, the time complexity of DeLong's algorithm is of quadratic order (the product of sample sizes), thus making it time-consuming and impractical when the sample sizes are large. Based on an equivalent relationship between the Heaviside function and mid-ranks of samples, we improve DeLong's algorithm by reducing the order of time complexity from quadratic down to linearithmic (the product of sample size and its logarithm). Monte Carlo simulations verify the computational efficiency of our algorithmic findings in this work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The hydrostats.ens metrics contains many commonly used metrics to gauge forecast skill for stochastic modeling, such as the ensemble mean squared error (EMSE), the continuous ranked probability score (CRPS) [23,24], the area under the receiver operating characteristic curve (AUROC) [25,26], and the ensemble adjusted brier score (EABS) [24].”
From Hydrostats: A Python Package for Characterizing Errors between Observed and Predicted Time Series (2018) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · cited 2×
“…the “structural components” and (8) Then, DeLong et al. [14] proposed a variance-covariance matrix estimator for the vector ˆ θθθ in (4), as When the vector ˆ θθθ contains only one element, that is, r = s = 1 in (7) and (8), the covariance estimator in (9) reduces to a variance estimator V (ˆ θ ) .”
BackgroundMethodThe meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
Grossman, P. (2009) · Discrete Mathematics for Computing · doi:10.1007/978-0-230-37405-8_1 · Cited by 8,834
Hanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016
Statistical Decision Functions
Wald, A. (1949) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730030 · Cited by 2,176
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
The theory of signal detectability
Peterson, W., Birdsall, T., & Fox, W. (1954) · Transactions of the IRE Professional Group on Information Theory · doi:10.1109/tit.1954.1057460 · Cited by 867
Receiver operating characteristic (ROC) methodology: the state of the art.
Hanley, J. A. (1989) · PubMed · Cited by 807
Nonparametrics: Statistical Methods Based on Ranks.
Berengut, D., & Lehmann, E. L. (1978) · Journal of the American Statistical Association · doi:10.2307/2286308 · Cited by 595
Cited by
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Vanguri, R. S., Luo, J., Aukerman, A. T., Egger, J. V., Fong, C. J., Horvat, N., Pagano, A., Araujo-Filho, J. D. A. B., Geneslaw, L., Rizvi, H., et al. (2022) · Nature Cancer · doi:10.1038/s43018-022-00416-8 · Cited by 395
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392
“For each pathology, we also test whether the AUC of the best-performing approach is significantly greater than the AUC of the worst-performing approach us-ing the one-sided DeLongs test for two correlated ROC curves (DeLong, DeLong, and Clarke-Pearson 1988).”
MethodPredicting HLA class II antigen presentation through integrated deep learning
Chen, B., Khodadoust, M. S., Olsson, N., Wagar, L. E., Fast, E., Liu, C. L., Muftuoglu, Y., Sworder, B. J., Diehn, M., Levy, R., et al. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0280-2 · Cited by 314
The Effect of Image Resolution on Deep Learning in Radiography
Sabottke, C. F., & Spieler, B. M. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2019190015 · Cited by 312
The False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184
Adedinsewo, D., Carter, R. E., Attia, Z., Johnson, P., Kashou, A. H., Dugan, J. L., Albus, M., Sheele, J. M., Bellolio, F., Friedman, P. A., et al. (2020) · Circulation: Arrhythmia and Electrophysiology · doi:10.1161/circep.120.008437 · Cited by 169
Rissman, R. A., Langford, O., Raman, R., Donohue, M. C., Abdel‐Latif, S., Meyer, M. R., Wente‐Roth, T., Kirmess, K. M., Ngolab, J., Winston, C. N., et al. (2024) · Alzheimer's & Dementia · doi:10.1002/alz.13542 · Cited by 157
Detection of Left Ventricular Systolic Dysfunction From Electrocardiographic Images
Sangha, V., Nargesi, A. A., Dhingra, L. S., Khunte, A., Mortazavi, B. J., Ribeiro, A. H., Banina, E., Adeola, O., Garg, N., Brandt, C. A., et al. (2023) · Circulation · doi:10.1161/circulationaha.122.062646 · Cited by 150
Lu, M. T., Raghu, V. K., Mayrhofer, T., Aerts, H. J. W. L., & Hoffmann, U. (2020) · Annals of Internal Medicine · doi:10.7326/m20-1868 · Cited by 123
“The DeLong method was used to compute 95% CIs and P values for comparison of correlated AUC curves (33).”
MethodSelf-supervised driven consistency training for annotation efficient histopathology image analysis
Srinidhi, C. L., Kim, S. W., Chen, F.-D., & Martel, A. L. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102256 · Cited by 116
Khunte, A., Sangha, V., Oikonomou, E. K., Dhingra, L. S., Aminorroaya, A., Mortazavi, B. J., Coppi, A., Brandt, C. A., Krumholz, H. M., & Khera, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00869-w · Cited by 95
Hydrostats: A Python Package for Characterizing Errors between Observed and Predicted Time Series
Roberts, W., Williams, G. P., Jackson, E., Nelson, E. J., & Ames, D. P. (2018) · Hydrology · doi:10.3390/hydrology5040066 · Cited by 85
Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification
Wollek, A., Graf, R., Čečatka, S., Fink, N., Willem, T., Sabel, B. O., & Lasser, T. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220187 · Cited by 68
Joseph, J. W., Leventhal, E. L., Grossestreuer, A. V., Wong, M. L., Joseph, L. J., Nathanson, L. A., Donnino, M. W., Elhadad, N., & Sanchez, L. D. (2020) · doi:10.1101/2020.05.02.20089052 · Cited by 57
Soltan, A. A. S., Yang, J., Pattanshetty, R., Novak, A., Yang, Y., Rohanian, O., Beer, S., Soltan, M. A., Thickett, D. R., Fairhead, R., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00272-7 · Cited by 54
“32 During prospective and external evaluation, we calculated and reported the proportions of missing data.”
A remote speech-based AI system to screen for early Alzheimer's disease via smartphones
Fristed, E., Skirrow, C., Meszaros, M., Lenain, R., Meepegama, U., Cappa, S., Aarsland, D., & Weston, J. (2022) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1002/dad2.12366 · Cited by 47
Dong, F., She, R., Cui, C., Shi, S., Hu, X., Zeng, J., Wu, H., Xu, J., & Zhang, Y. (2021) · European Radiology · doi:10.1007/s00330-020-07561-7 · Cited by 43
Identification of hypertrophic cardiomyopathy on electrocardiographic images with deep learning
Sangha, V., Dhingra, L. S., Aminorroaya, A., Croon, P. M., Sikand, N. V., Sen, S., Martinez, M. W., Maron, M. S., Krumholz, H. M., Asselbergs, F. W., et al. (2025) · Nature Cardiovascular Research · doi:10.1038/s44161-025-00685-3 · Cited by 41
Data consistency in the English Hospital Episodes Statistics database
Hardy, F., Heyl, J., Tucker, K., Hopper, A., Marchã, M. J., Briggs, T. W. R., Yates, J., Day, J., Wheeler, A., Eve-Jones, S., et al. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2022-100633 · Cited by 38
Identification of hypertrophic cardiomyopathy on electrocardiographic images with deep learning
Sangha, V., Dhingra, L. S., Aminorroaya, A., Croon, P. M., Sikand, N. V., Sen, S., Martinez, M. W., Maron, M. S., Krumholz, H. M., Asselbergs, F. W., et al. (2023) · doi:10.1101/2023.12.23.23300490 · Cited by 38
Hinson, J. S., Klein, E., Smith, A., Toerper, M., Dungarani, T., Hager, D., Hill, P., Kelen, G., Niforatos, J. D., Stephens, R. S., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00646-1 · Cited by 30
Vodrahalli, K., Ko, J., Chiou, A. S., Novoa, R., Abid, A., Phung, M., Yekrang, K., Petrone, P., Zou, J., & Daneshjou, R. (2023) · JAMA Dermatology · doi:10.1001/jamadermatol.2023.0091 · Cited by 30
Fristed, E., Skirrow, C., Meszaros, M., Lenain, R., Meepegama, U., Papp, K. V., Ropacki, M., & Weston, J. (2022) · Brain Communications · doi:10.1093/braincomms/fcac231 · Cited by 29
Chiu, Y. M., Courteau, J., Dufour, I., Vanasse, A., & Hudon, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-27568-6 · Cited by 20
“AUC 95% confidence intervals were also computed using DeLong’s method 44 .”
MethodStacking machine learning algorithms for biomarker-based preoperative diagnosis of a pelvic mass
Shaw, R., Lokshin, A. E., Miller, M. C., Messerlian-Lambert, G., & Moore, R. G. (2022) · Cancers · doi:10.3390/cancers14051291 · Cited by 16
Holistic AI analysis of hybrid cardiac perfusion images for mortality prediction
Marcinkiewicz, A. M., Zhang, W., Shanbhag, A., Miller, R. J. H., Lemley, M., Ramirez, G., Buchwald, M., Killekar, A., Kavanagh, P. B., Feher, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01526-0 · Cited by 12
“…of information among data splits 41 ; 3) ensuring each data point contributes to the test set exactly once, providing independent and non-overlapping predictions for robust performance evaluation 42 ; 4) meetingtheDeLong testrequirementsforvalid AUCcomparisonsbyusing independent predictions 43 .”
Multi-cohort machine learning identifies predictors of cognitive impairment in Parkinson’s disease
Loo, R. T. J., Pavelka, L., Mangone, G., Khoury, F., Vidailhet, M., Corvol, J.-C., Glaab, E., On behalf of the NCER-PD Consortium, Acharya, G., Aguayo, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01862-1 · Cited by 9
Bäcklund, N., Brattsand, G., Lundstedt, S., Aardal, E., Bartuseviciene, I., Berinder, K., Höybye, C., Burman, P., Edén Engström, B., Isaksson, A., et al. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2023-0141 · Cited by 9
Robust Speech and Natural Language Processing Models for Depression Screening
Lu, Y., Harati, A., Rutowski, T., Oliveira, R., Chlebek, P., & Shriberg, E. (2020) · 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB) · doi:10.1109/spmb50085.2020.9353611 · Cited by 9
Kishikawa, R., Kodera, S., Setoguchi, N., Tanabe, K., Kushida, S., Nanasato, M., Maki, H., Fujita, H., Kato, N., Watanabe, H., et al. (2025) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae097 · Cited by 8
Ajuwon, B. I., Richardson, A., Roper, K., & Lidbury, B. A. (2023) · Viruses · doi:10.3390/v15081735 · Cited by 6
Metadata
- Authors
- Sun, X., & Xu, W.
- Venue
- IEEE Signal Processing Letters
- Published
- 2014-11
- Volume
- 21
- Issue
- 11
- Pages
- 1389-1393
- ISSN
- 1070-9908 (ISSN-L)
- 1558-2361
- OpenAlex
- W2042571564
- Cited by
- 630
- References
- 22
- FWCI
- 2.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.