The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
The Inconsistency of “Optimal” Cutpoints Obtained using Two Criteria based on the Receiver Operating Characteristic Curve
Open accessAbstract
The use of biomarkers is of ever-increasing importance in clinical diagnosis of disease. In practice, a cutpoint is required for dichotomizing naturally continuous biomarker levels to distinguish persons at risk of disease from those who are not. Two methods commonly used for establishing the "optimal" cutpoint are the point on the receiver operating characteristic curve closest to (0,1) and the Youden index, J. Both have sound intuitive interpretations--the point closest to perfect differentiation and the point farthest from none, respectively--and are generalizable to weighted sensitivity and specificity. Under the same weighting of sensitivity and specificity, these two methods identify the same cutpoint as "optimal" in certain situations but different cutpoints in others. In this paper, the authors examine situations in which the two criteria agree or disagree and show that J is the only "optimal" cutpoint for given weighting with respect to overall misclassification rates. A data-driven example is used to clarify and demonstrate the magnitude of the differences. The authors also demonstrate a slight alteration in the (0,1) criterion that retains its intuitive meaning while resulting in consistent agreement with J. In conclusion, the authors urge that great care be taken when establishing a biomarker cutpoint for clinical use.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
Zweig, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,569
Circulating angiogenic factors and the risk of preeclampsia
Levine, R. J., Maynard, S. E., Qian, C., Lim, K.-H., England, L. J., Yu, K. F., Schisterman, E. F., Thadhani, R., Sachs, B. P., Epstein, F. H., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejmoa031884 · Cited by 3,720
Self-citationCirculating angiogenic factors and the risk of preeclampsia*
(2004) · Obstetrics & Gynecology · doi:10.1097/00006250-200405000-00028 · Cited by 2,615
The Statistical Evaluation of Medical Tests for Classification and Prediction
Pepe, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,198
Greiner, M., Pfeiffer, D., & Smith, R. D. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00115-x · Cited by 1,923
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Statistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470906514 · Cited by 1,784
Schisterman, E. F., Perkins, N. J., Liu, A., & Bondell, H. (2005) · Epidemiology · doi:10.1097/01.ede.0000147512.81966.ba · Cited by 1,185
Self-citationStatistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9780470317082 · Cited by 1,165
Evaluating Medical Tests: Objective and Quantitative Guidelines.
Guthrie, D., & Kraemer, H. C. (1992) · Journal of the American Statistical Association · doi:10.2307/2290671 · Cited by 629
Cited by
Seymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0288 · Cited by 3,939
“(%) 17 327 (12) 2153 (27) 6517 (10) 2171 (27) 6486 (10)”
(2013) · Europe PMC (PubMed Central) · Cited by 2,089
Understanding diagnostic tests 3: receiver operating characteristic curves
Akobeng, A. K. (2007) · Acta Paediatrica · doi:10.1111/j.1651-2227.2006.00178.x · Cited by 1,882
Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators
Van Calster, B., Wynants, L., Verbeek, J. F. M., Verbakel, J. Y., Christodoulou, E., Vickers, A. J., Roobol, M. J., & Steyerberg, E. W. (2018) · European Urology · doi:10.1016/j.eururo.2018.08.038 · Cited by 1,271
Budczies, J., Klauschen, F., Sinn, B. V., Győrffy, B., Schmitt, W. D., Darb-Esfahani, S., & Denkert, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051862 · Cited by 1,224
Receiver operating characteristic (ROC) curve for medical researchers
Kumar, R., & Indrayan, A. (2011) · Indian Pediatrics · doi:10.1007/s13312-011-0055-4 · Cited by 872
Machado, P., Landewé, R., Lie, E., Kvien, T. K., Braun, J., Baker, D., & van der Heijde, D. (2011) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2010.138594 · Cited by 805
Jiménez‐Valverde, A. (2012) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2011.00683.x · Cited by 803
Defining an Optimal Cut-Point Value in ROC Analysis: An Alternative Approach
Unal, I. (2017) · Computational and Mathematical Methods in Medicine · doi:10.1155/2017/3762651 · Cited by 789
On determining the most appropriate test cut-off value: the case of tests with continuous results
Habibzadeh, F., Habibzadeh, P., & Yadollahie, M. (2016) · Biochemia Medica · doi:10.11613/bm.2016.034 · Cited by 694
Classification accuracy and cut point selection
Liu, X. (2012) · Statistics in Medicine · doi:10.1002/sim.4509 · Cited by 676
Defining hematoma expansion in intracerebral hemorrhage: Relationship with patient outcomes
Dowlatshahi, D., Demchuk, A. M., Flaherty, M. L., Ali, M., Lyden, P. L., & Smith, E. E. (2011) · Neurology · doi:10.1212/wnl.0b013e3182143317 · Cited by 641
Validation of the GENEA Accelerometer
ESLIGER, D. W., ROWLANDS, A. V., HURST, T. L., CATT, M., MURRAY, P., & ESTON, R. G. (2011) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31820513be · Cited by 625
OptimalCutpoints: An R Package for Selecting Optimal Cutpoints in Diagnostic Tests
López-Ratón, M., Rodríguez-Álvarez, M. X., Suárez, C. C., & Sampedro, F. G. (2014) · Journal of Statistical Software · doi:10.18637/jss.v061.i08 · Cited by 548
Statistical Evaluation of a Biomarker
Ray, P., Manach, Y. L., Riou, B., Houle, T. T., & Warner, D. S. (2010) · Anesthesiology · doi:10.1097/aln.0b013e3181d47604 · Cited by 503
Validation of a Korean Version of the Insomnia Severity Index
Cho, Y. W., Song, M. L., & Morinc, C. M. (2014) · Journal of Clinical Neurology · doi:10.3988/jcn.2014.10.3.210 · Cited by 472
Yumino, D., Redolfi, S., Ruttanaumpawan, P., Su, M.-C., Smith, S., Newton, G. E., Mak, S., & Bradley, T. D. (2010) · Circulation · doi:10.1161/circulationaha.109.902452 · Cited by 452
Translation of proteomic biomarkers into FDA approved cancer diagnostics: issues and challenges
Füzéry, A. K., Levin, J., Chan, M. M., & Chan, D. W. (2013) · Clinical Proteomics · doi:10.1186/1559-0275-10-13 · Cited by 428
Evaluation of raw acceleration sedentary thresholds in children and adults
Hildebrand, M., Hansen, B. H., van Hees, V. T., & Ekelund, U. (2017) · Scandinavian Journal of Medicine & Science in Sports · doi:10.1111/sms.12795 · Cited by 425
Tzankov, A., Meier, C., Hirschmann, P., Went, P., Pileri, S. A., & Dirnhofer, S. (2008) · Haematologica · doi:10.3324/haematol.11702 · Cited by 412
Kawakami, R., Murakami, H., Sanada, K., Tanaka, N., Sawada, S. S., Tabata, I., Higuchi, M., & Miyachi, M. (2015) · Geriatrics & Gerontology International · doi:10.1111/ggi.12377 · Cited by 392
Predicting the conservation status of data-deficient species
Bland, L. M., Collen, B., Orme, C. D. L., & Bielby, J. (2015) · Conservation Biology · doi:10.1111/cobi.12372 · Cited by 364
The Triglyceride-to-HDL Cholesterol Ratio
Giannini, C., Santoro, N., Caprio, S., Kim, G., Lartaud, D., Shaw, M., Pierpont, B., & Weiss, R. (2011) · Diabetes Care · doi:10.2337/dc10-2234 · Cited by 360
Button, K. S., Kounali, D., Thomas, L., Wiles, N. J., Peters, T. J., Welton, N. J., Ades, A. E., & Lewis, G. (2015) · Psychological Medicine · doi:10.1017/s0033291715001270 · Cited by 332
Harpaz, R., DuMouchel, W., LePendu, P., Bauer-Mehren, A., Ryan, P., & Shah, N. H. (2013) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2013.24 · Cited by 333
Moskowitz, C. H., Schöder, H., Teruya-Feldstein, J., Sima, C., Iasonos, A., Portlock, C. S., Straus, D., Noy, A., Palomba, M. L., O'Connor, O. A., et al. (2010) · Journal of Clinical Oncology · doi:10.1200/jco.2009.26.5942 · Cited by 326
Pan, D., Kocherginsky, M., & Conzen, S. D. (2011) · Cancer Research · doi:10.1158/0008-5472.can-11-0362 · Cited by 296
Hansson, O., Zetterberg, H., Buchhave, P., Andreasson, U., Londos, E., Minthon, L., & Blennow, K. (2007) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000100926 · Cited by 288
The Performance of Risk Prediction Models
Gerds, T. A., Cai, T., & Schumacher, M. (2008) · Biometrical Journal · doi:10.1002/bimj.200810443 · Cited by 288
Self, W. H., Williams, D. J., Zhu, Y., Ampofo, K., Pavia, A. T., Chappell, J. D., Hymas, W. C., Stockmann, C., Bramley, A. M., Schneider, E., et al. (2016) · Journal of Infectious Diseases · doi:10.1093/infdis/jiv323 · Cited by 273
Ravi, B., Jenkinson, R., Austin, P. C., Croxford, R., Wasserstein, D., Escott, B., Paterson, J. M., Kreder, H., & Hawker, G. A. (2014) · BMJ · doi:10.1136/bmj.g3284 · Cited by 270
Robust Early Pregnancy Prediction of Later Preeclampsia Using Metabolomic Biomarkers
Kenny, L. C., Broadhurst, D. I., Dunn, W., Brown, M., North, R. A., McCowan, L., Roberts, C., Cooper, G. J. S., Kell, D. B., & Baker, P. N. (2010) · Hypertension · doi:10.1161/hypertensionaha.110.157297 · Cited by 269
Circulating microRNAs as Specific Biomarkers for Breast Cancer Detection
Ng, E. K. O., Li, R., Shin, V. Y., Jin, H. C., Leung, C. P. H., Ma, E. S. K., Pang, R., Chua, D., Chu, K.-M., Law, W. L., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0053141 · Cited by 263
Optimal cutoffs for the Montreal Cognitive Assessment vary by race and ethnicity
Milani, S. A., Marsiske, M., Cottler, L. B., Chen, X., & Striley, C. W. (2018) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2018.09.003 · Cited by 259
Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning
Liu, D., Zhao, Y., Xu, H., Sun, Y., Pei, D., Luo, J., Jing, X., & Feng, M. (2015) · Proceedings of the 2015 Internet Measurement Conference · doi:10.1145/2815675.2815679 · Cited by 252
Physical Activity, Sedentary Time, and Obesity in an International Sample of Children
KATZMARZYK, P. T., BARREIRA, T. V., BROYLES, S. T., CHAMPAGNE, C. M., CHAPUT, J.-P., FOGELHOLM, M., HU, G., JOHNSON, W. D., KURIYAN, R., KURPAD, A., et al. (2015) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000000649 · Cited by 222
“The optimal threshold was determined from the Youden index (21), which is the maximum value of J:”
BackgroundEsteghamati, A., Ashraf, H., Khalilzadeh, O., Zandieh, A., Nakhjavani, M., Rashidi, A., Haghazali, M., & Asgari, F. (2010) · Nutrition & Metabolism · doi:10.1186/1743-7075-7-26 · Cited by 212
Hassanzad, M., & Hajian-Tilaki, K. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02198-2 · Cited by 208
Younes, R., Caviglia, G. P., Govaere, O., Rosso, C., Armandi, A., Sanavia, T., Pennisi, G., Liguori, A., Francione, P., Gallego-Durán, R., et al. (2021) · Journal of Hepatology · doi:10.1016/j.jhep.2021.05.008 · Cited by 196
Patterns and prognostic relevance of PD-1 and PD-L1 expression in colorectal carcinoma
Lee, L. H., Cavalcanti, M. S., Segal, N. H., Hechtman, J. F., Weiser, M. R., Smith, J. J., Garcia-Aguilar, J., Sadot, E., Ntiamoah, P., Markowitz, A. J., et al. (2016) · Modern Pathology · doi:10.1038/modpathol.2016.139 · Cited by 186
Yanamandra, K., Gruden, M. A., Casaite, V., Meskys, R., Forsgren, L., & Morozova-Roche, L. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018513 · Cited by 171
“J corresponds to the maximum vertical distance between the ROC curve and the diagonal or chance line and it occurs at the cut-off point when equal weight is given to sensitivity and specificity [63,64].”
BackgroundVerbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.028 · Cited by 138
Zhang, L., Zhang, Z., Zhang, Y., Hu, G., & Chen, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0097865 · Cited by 130
Angulo, M., Rooks, B. T., Gill, M., Goldstein, T., Sakolsky, D., Goldstein, B., Monk, K., Hickey, M. B., Diler, R. S., Hafeman, D., et al. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2017.02.034 · Cited by 128
“The Euclidean method (Perkins and Schisterman, 2006) finds the cutoff of the classifier where the distance between the ROC curve and the upper left corner (0,1) (which corresponds to perfect prediction) is minimized.”
MethodTakala, R. S. K., Posti, J. P., Runtti, H., Newcombe, V. F., Outtrim, J., Katila, A. J., Frantzén, J., Ala-Seppälä, H., Kyllönen, A., Maanpää, H.-R., et al. (2016) · World Neurosurgery · doi:10.1016/j.wneu.2015.10.066 · Cited by 105
Body Composition in Asians and Caucasians
Haldar, S., Chia, S. C., & Henry, C. J. (2015) · Advances in Food and Nutrition Research · doi:10.1016/bs.afnr.2015.07.001 · Cited by 100
Woolcott, O. O., & Bergman, R. N. (2020) · International Journal of Obesity · doi:10.1038/s41366-019-0516-8 · Cited by 97
HbA1c Measured in the First Trimester of Pregnancy and the Association with Gestational Diabetes
Hinkle, S. N., Tsai, M. Y., Rawal, S., Albert, P. S., & Zhang, C. (2018) · Scientific Reports · doi:10.1038/s41598-018-30833-8 · Cited by 57
A Single-Item Visual Analogue Scale (VAS) Measure for Assessing Depression among College Students
Huang, Z., Kohler, I. V., & Kämpfen, F. (2020) · Community Mental Health Journal · doi:10.1007/s10597-019-00469-7 · Cited by 51
Dahlgren, C. L., Hage, T. W., Wonderlich, J. A., & Stedal, K. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00663 · Cited by 40
Identifying standardized definitions of treatment response in trichotillomania: A meta-analysis
Farhat, L. C., Olfson, E., Li, F., Telang, S., & Bloch, M. H. (2019) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2018.10.009 · Cited by 32
Tzikos, G., Alexiou, I., Tsagkaropoulos, S., Menni, A.-E., Chatziantoniou, G., Doutsini, S., Papavramidis, T., Grosomanidis, V., Stavrou, G., & Kotzampassi, K. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13030473 · Cited by 28
AI Advances in ICU with an Emphasis on Sepsis Prediction: An Overview
Stylianides, C., Nicolaou, A., Sulaiman, W. A., Alexandropoulou, C.-A., Panagiotopoulos, I., Karathanasopoulou, K., Dimitrakopoulos, G., Kleanthous, S., Politi, E., Ntalaperas, D., et al. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7010006 · Cited by 28
“The threshold is ‘optimal’ when it classifies most of the individuals correctly [212], ensuring maximum sensitivity and specificity.”
Can artificial intelligence simplify the screening of muscle mass loss?
Buccheri, E., Dell’Aquila, D., Russo, M., Chiaramonte, R., Musumeci, G., & Vecchio, M. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16323 · Cited by 27
Landsheer, J. A. (2018) · Diagnostics · doi:10.3390/diagnostics8020032 · Cited by 27
Sesso, G., Milone, A., Drago, F., Viglione, V., Berloffa, S., Boldrini, S., Loriaux, N., Valente, E., Molesti, A., Placini, F., et al. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.05.037 · Cited by 26
Buccheri, E., Dell'Aquila, D., & Russo, M. (2021) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2021.108722 · Cited by 24
A Diagnostic Biomarker for Pediatric Generalized Anxiety Disorder Using the Error-Related Negativity
Hanna, G. L., Liu, Y., Rough, H. E., Surapaneni, M., Hanna, B. S., Arnold, P. D., & Gehring, W. J. (2020) · Child Psychiatry & Human Development · doi:10.1007/s10578-020-01021-5 · Cited by 24
CHOU, K., CHANG, Y., CHEN, Y., SU, K., PERNG, D., CHANG, S., & SHIAO, G. (2011) · Respirology · doi:10.1111/j.1440-1843.2011.02022.x · Cited by 18
Development of a brief parent-report risk index for children following parental divorce
Tein, J.-Y., Sandler, I. N., Braver, S. L., & Wolchik, S. A. (2013) · Journal of Family Psychology · doi:10.1037/a0034571 · Cited by 16
“This criteria selects the cut-point as the point on the curve closest to (0, 1) in the ROC curve (Perkins & Schisterman, 2006).”
BackgroundYavropoulou, M. P., Filippa, M. G., Vlachogiannis, N. I., Fragoulis, G. E., Laskari, K., Mantzou, A., Panopoulos, S., Fanouriakis, A., Bournia, V.-K., Evangelatos, G., et al. (2024) · RMD Open · doi:10.1136/rmdopen-2023-003575 · Cited by 13
“We used the point closest to (0.1) corner in the ROC curve 33 to define the optimal cut-off point value.”
InfluentialSilva, A. I., Camelo, A., Madureira, J., Reis, A. T., Machado, A. P., Teixeira, J. P., & Costa, C. (2022) · Environmental Research · doi:10.1016/j.envres.2021.111827 · Cited by 13
Combined effects of social isolation and loneliness on disability incidence in older adults
Shimada, H., Doi, T., Tsutsumimoto, K., Makino, K., Harada, K., Tomida, K., Morikawa, M., & Arai, H. (2025) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2025.105749 · Cited by 9
Rehman, H., Initiative, F. T. A. D. N., Ang, T. F. A., Tao, Q., Espenilla, A. L., Au, R., Farrer, L. A., Zhang, X., & Qiu, W. Q. (2024) · Journal of Alzheimer's Disease · doi:10.3233/jad-230837 · Cited by 7
Ultrasonographic diagnosis of clinical and subclinical bovine respiratory disease in Holstein calves
Mahmoud, A. E., Fathy, A., Ahmed, E. A., Ali, A. O., Abdelaal, A. M., & El-Maghraby, M. M. (2022) · Veterinary World · doi:10.14202/vetworld.2022.1932-1942 · Cited by 6
“The second method is the most commonly used one [22].”
Method
Metadata
- Authors
- Perkins, N. J., & Schisterman, E. F.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-01-12
- Volume
- 163
- Issue
- 7
- Pages
- 670-675
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2117352205
- Cited by
- 1,759
- References
- 18
- FWCI
- 24.10
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.