The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Bias in Sensitivity and Specificity Caused by Data-Driven Selection of Optimal Cutoff Values: Mechanisms, Magnitude, and Solutions
Abstract
Background: Optimal cutoff values for tests results involving continuous variables are often derived in a data-driven way. This approach, however, may lead to overly optimistic measures of diagnostic accuracy. We evaluated the magnitude of the bias in sensitivity and specificity associated with data-driven selection of cutoff values and examined potential solutions to reduce this bias. Methods: Different sample sizes, distributions, and prevalences were used in a simulation study. We compared data-driven estimates of accuracy based on the Youden index with the true values and calculated the median bias. Three alternative approaches (assuming a specific distribution, leave-one-out, smoothed ROC curve) were examined for their ability to reduce this bias. Results: The magnitude of bias caused by data-driven optimization of cutoff values was inversely related to sample size. If the true values for sensitivity and specificity are both 84%, the estimates in studies with a sample size of 40 will be approximately 90%. If the sample size increases to 200, the estimates will be 86%. The distribution of the test results had little impact on the amount of bias when sample size was held constant. More robust methods of optimizing cutoff values were less prone to bias, but the performance deteriorated if the underlying assumptions were not met. Conclusions: Data-driven selection of the optimal cutoff value can lead to overly optimistic estimates of sensitivity and specificity, especially in small studies. Alternative methods can reduce this bias, but finding robust estimates for cutoff values and accuracy requires considerable sample sizes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimation of the Youden Index and its Associated Cutoff Point
Fluss, R., Faraggi, D., & Reiser, B. (2005) · Biometrical Journal · doi:10.1002/bimj.200410135 · Cited by 2,608
Greiner, M., Pfeiffer, D., & Smith, R. D. (2000) · Preventive Veterinary Medicine · doi:10.1016/s0167-5877(00)00115-x · Cited by 1,923
Schisterman, E. F., Perkins, N. J., Liu, A., & Bondell, H. (2005) · Epidemiology · doi:10.1097/01.ede.0000147512.81966.ba · Cited by 1,185
Dangers of Using "Optimal" Cutpoints in the Evaluation of Prognostic Factors
Altman, D. G., Lausen, B., Sauerbrei, W., & Schumacher, M. (1994) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/86.11.829 · Cited by 1,143
The interpretation of diagnostic tests
Shapiro, D. E. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800203 · Cited by 383
ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions
Obuchowski, N. A., Lieber, M. L., & Wians, F. H. (2004) · Clinical Chemistry · doi:10.1373/clinchem.2004.031823 · Cited by 358
The Youden Index and the Optimal Cut‐Point Corrected for Measurement Error
Perkins, N. J., & Schisterman, E. F. (2005) · Biometrical Journal · doi:10.1002/bimj.200410133 · Cited by 288
Sample sizes of studies on diagnostic accuracy: literature survey.
Bachmann, L. M., Puhan, M. A., Riet, G. T., & Bossuyt, P. M. (2006) · BMJ · doi:10.1136/bmj.38793.637789.2f · Cited by 184
Post hoc choice of cut points introduced bias to diagnostic research
Ewald, B. D. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.11.025 · Cited by 94
A solution for the most basic optimization problem associated with an ROC curve
Le, C. T. (2006) · Statistical Methods in Medical Research · doi:10.1177/0962280206070637 · Cited by 85
Cited by
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citationSTARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
“The reporting of performance based on thresholds chosen from the data itself can produce over-optimistic and biased performance [95].”
BackgroundSelf-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationSystematic Reviews of Diagnostic Test Accuracy
Leeflang, M. M. G., Deeks, J. J., Gatsonis, C., & Bossuyt, P. M. M. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-149-12-200812160-00008 · Cited by 1,124
Self-citation
Metadata
- Authors
- Leeflang, M. M. G., Moons, K. G. M., Reitsma, J. B., & Zwinderman, A. H.
- Venue
- Clinical Chemistry
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-04-01
- Volume
- 54
- Issue
- 4
- Pages
- 729-737
- ISSN
- 0009-9147 (ISSN-L)
- 1530-8561
- OpenAlex
- W2168153432
- Cited by
- 346
- References
- 13
- FWCI
- 8.38
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bias in Sensitivity and Specificity Caused by Data-Driven Selection of Optimal Cutoff Values: Mechanisms, Magnitude, and Solutions. https://app.cochat.ai/papers/10.1373/clinchem.2007.096032
- The paper
- Leeflang, M. M. G., Moons, K. G. M., Reitsma, J. B., & Zwinderman, A. H. (2008). Bias in Sensitivity and Specificity Caused by Data-Driven Selection of Optimal Cutoff Values: Mechanisms, Magnitude, and Solutions. Clinical Chemistry. https://doi.org/10.1373/clinchem.2007.096032
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
