The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves
Abstract
Among algorithms for comparing the areas under two or more correlated receiver operating characteristic (ROC) curves, DeLong's algorithm is perhaps the most widely used one due to its simplicity of implementation in practice. Unfortunately, however, the time complexity of DeLong's algorithm is of quadratic order (the product of sample sizes), thus making it time-consuming and impractical when the sample sizes are large. Based on an equivalent relationship between the Heaviside function and mid-ranks of samples, we improve DeLong's algorithm by reducing the order of time complexity from quadratic down to linearithmic (the product of sample size and its logarithm). Monte Carlo simulations verify the computational efficiency of our algorithmic findings in this work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · cited 2×
“…the “structural components” and (8) Then, DeLong et al. [14] proposed a variance-covariance matrix estimator for the vector ˆ θθθ in (4), as When the vector ˆ θθθ contains only one element, that is, r = s = 1 in (7) and (8), the covariance estimator in (9) reduces to a variance estimator V (ˆ θ ) .”
BackgroundMethodThe meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
Grossman, P. (2009) · Discrete Mathematics for Computing · doi:10.1007/978-0-230-37405-8_1 · Cited by 8,834
Hanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016
Statistical Decision Functions
Wald, A. (1949) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730030 · Cited by 2,176
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
The theory of signal detectability
Peterson, W., Birdsall, T., & Fox, W. (1954) · Transactions of the IRE Professional Group on Information Theory · doi:10.1109/tit.1954.1057460 · Cited by 867
Receiver operating characteristic (ROC) methodology: the state of the art.
Hanley, J. A. (1989) · PubMed · Cited by 685
Nonparametrics: Statistical Methods Based on Ranks.
Berengut, D., & Lehmann, E. L. (1978) · Journal of the American Statistical Association · doi:10.2307/2286308 · Cited by 595
Farlie, D. J. G., & Kendall, M. G. (1971) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2343668 · Cited by 344
Mossman, D. (1999) · Medical Decision Making · doi:10.1177/0272989x9901900110 · Cited by 302
Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests
Zou, K. H., Hall, W. J., & Shapiro, D. E. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19971015)16:19<2143::aid-sim655>3.0.co;2-3 · Cited by 274
A comparative analysis of Spearman's rho and Kendall's tau in normal and contaminated normal models
Xu, W., Hou, Y., Hung, Y. S., & Zou, Y. (2013) · Signal Processing · doi:10.1016/j.sigpro.2012.08.005 · Cited by 181
Self-citationModern statistical approaches to reception in communication theory
Meter, D. V., & Middleton, D. (1954) · Transactions of the IRE Professional Group on Information Theory · doi:10.1109/tit.1954.1057471 · Cited by 95
Design and performance analysis of a signal detector based on suprathreshold stochastic resonance
Vishnu, H., Anand, G. V., Premkumar, A. B., & Madhukumar, A. S. (2012) · Signal Processing · doi:10.1016/j.sigpro.2012.01.013 · Cited by 61
Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach
Yousef, W. A., Wagner, R. F., & Loew, M. H. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.218 · Cited by 43
Asymptotic Mean and Variance of Gini Correlation for Bivariate Normal Samples
Xu, W., Hung, Y. S., Niranjan, M., & Shen, M. (2009) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2009.2032448 · Cited by 40
Self-citationAsymptotic Properties of Order Statistics Correlation Coefficient in the Normal Cases
Xu, W., Chang, C., Hung, Y. S., & Fung, P. C. W. (2008) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2007.916127 · Cited by 37
Self-citationNonparametric ROC and LROC analysis
Popescu, L. M. (2007) · Medical Physics · doi:10.1118/1.2717407 · Cited by 35
A Nonparametric Procedure for Comparing the Areas Under Correlated LROC Curves
Wunderlich, A., & Noo, F. (2012) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2012.2205015 · Cited by 21
Oukaci, A., Delmas, J.-P., & Chevalier, P. (2011) · Signal Processing · doi:10.1016/j.sigpro.2011.04.017 · Cited by 9
Cited by
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Vanguri, R. S., Luo, J., Aukerman, A. T., Egger, J. V., Fong, C. J., Horvat, N., Pagano, A., Araujo-Filho, J. D. A. B., Geneslaw, L., Rizvi, H., et al. (2022) · Nature Cancer · doi:10.1038/s43018-022-00416-8 · Cited by 395
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392
“For each pathology, we also test whether the AUC of the best-performing approach is significantly greater than the AUC of the worst-performing approach us-ing the one-sided DeLongs test for two correlated ROC curves (DeLong, DeLong, and Clarke-Pearson 1988).”
MethodPredicting HLA class II antigen presentation through integrated deep learning
Chen, B., Khodadoust, M. S., Olsson, N., Wagar, L. E., Fast, E., Liu, C. L., Muftuoglu, Y., Sworder, B. J., Diehn, M., Levy, R., et al. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0280-2 · Cited by 314
Metadata
- Authors
- Sun, X., & Xu, W.
- Venue
- IEEE Signal Processing Letters
- Published
- 2014-11
- Volume
- 21
- Issue
- 11
- Pages
- 1389-1393
- ISSN
- 1070-9908 (ISSN-L)
- 1558-2361
- OpenAlex
- W2042571564
- Cited by
- 630
- References
- 22
- FWCI
- 2.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.