Calibration of medical diagnostic classifier scores to the probability of disease
Open accessAbstract
Scores produced by statistical classifiers in many clinical decision support systems and other medical diagnostic devices are generally on an arbitrary scale, so the clinical meaning of these scores is unclear. Calibration of classifier scores to a meaningful scale such as the probability of disease is potentially useful when such scores are used by a physician. In this work, we investigated three methods (parametric, semi-parametric, and non-parametric) for calibrating classifier scores to the probability of disease scale and developed uncertainty estimation techniques for these methods. We showed that classifier scores on arbitrary scales can be calibrated to the probability of disease scale without affecting their discrimination performance. With a finite dataset to train the calibration function, it is important to accompany the probability estimate with its confidence interval. Our simulations indicate that, when a dataset used for finding the transformation for calibration is also used for estimating the performance of calibration, the resubstitution bias exists for a performance metric involving the truth states in evaluating the calibration performance. However, the bias is small for the parametric and semi-parametric methods when the sample size is moderate to large (>100 per class).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The calibration function is required to be monotonically increasing in order to calibrate predictors without changing their ability to discriminate.
“As noted by Chen et al, 32 the calibration function is required to be monotonically increasing in order to calibrate predictors without changing their ability to discriminate.”
From Calibrating machine learning approaches for probability estimation: A comprehensive comparison (2023) · cites this paper as [32]
Chen calibration performed well in its original publication.
“Chen calibration performed well in their original publication.32”
From Calibrating machine learning approaches for probability estimation: A comprehensive comparison (2023) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Numerical recipes in C: the art of scientific computing
(1993) · Choice Reviews Online · doi:10.5860/choice.30-5638 · Cited by 17,933 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834 · see citing works
Gail, M. H., Brinton, L. A., Byar, D. P., Corle, D. K., Green, S. B., Schairer, C., & Mulvihill, J. J. (1989) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/81.24.1879 · Cited by 3,334 · see citing works
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917 · see citing works
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,206 · see citing works
Score normalization in multimodal biometric systems
Jain, A. K., Nandakumar, K., & Ross, A. A. (2005) · Pattern Recognition · doi:10.1016/j.patcog.2005.01.012 · Cited by 2,171 · see citing works
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,123 · see citing works
Cover, T. M. (1965) · IEEE Transactions on Electronic Computers · doi:10.1109/pgec.1965.264137 · Cited by 2,044 · see citing works
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,907 · see citing works
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · see citing works
Computer-aided diagnosis in medical imaging: historical review, current status and future potential
Doi, K. (2007) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2007.02.002 · Cited by 1,855 · see citing works
Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)
Rasmussen, C. E., & Williams, C. K. I. (2005) · The MIT Press eBooks · Cited by 1,829 · see citing works
Statistical Methods in Diagnostic Medicine
Zhou, X., Sun, J., Pennello, G. A., Obuchowski, N. A., & McClish, D. K. (2011) · Wiley series in probability and statistics · doi:10.1002/9780470906514 · Cited by 1,785 · see citing works
Signal detection theory and ROC analysis
Egan, J. P. (1975) · Academic Press eBooks · Cited by 1,656 · see citing works
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,458 · see citing works
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
Statistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2002) · Wiley series in probability and statistics · doi:10.1002/9780470317082 · Cited by 1,181 · see citing works
Platt, J. C. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0008 · Cited by 1,156 · see citing works
An Empirical Distribution Function for Sampling with Incomplete Information
Ayer, M., Brunk, H. D., Ewing, G. M., Reid, W. T., & Silverman, E. (1955) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728423 · Cited by 896 · see citing works
MAQC Consortium (2010) · Nature Biotechnology · doi:10.1038/nbt.1665 · Cited by 889 · see citing works
Self-citationA note on Platt’s probabilistic outputs for support vector machines
Lin, H.-T., Lin, C., & Weng, R. C. (2007) · Machine Learning · doi:10.1007/s10994-007-5018-6 · Cited by 876 · see citing works
Leening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594 · see citing works
Effects of sample size in classifier design
Fukunaga, K., & Hayes, R. R. (1989) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.31448 · Cited by 320 · see citing works
“Proper” Binormal ROC Curves: Theory and Maximum-Likelihood Estimation
Metz, C. E., & Pan, X. (1999) · Journal of Mathematical Psychology · doi:10.1006/jmps.1998.1218 · Cited by 275 · see citing works
Linear Combinations of Multiple Diagnostic Markers
Su, J. Q., & Liu, J. S. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476417 · Cited by 275 · see citing works
Hilden, J., & Gerds, T. A. (2013) · Statistics in Medicine · doi:10.1002/sim.5804 · Cited by 200 · see citing works
Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J. (2015) · Statistics in Biosciences · doi:10.1007/s12561-014-9118-0 · Cited by 191 · see citing works
What price perfection? Calibration and discrimination of clinical prediction models
Diamond, G. A. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90192-p · Cited by 155 · see citing works
Fawcett, T., & Niculescu-Mizil, A. (2007) · Machine Learning · doi:10.1007/s10994-007-5011-0 · Cited by 109 · see citing works
Medical Imaging 2014: Image Perception, Observer Performance, and Technology Assessment
Mello-Thoms, C. R., & Kupinski, M. A. (2014) · SPIE eBooks · Cited by 79 · see citing works
Shi, L. M., Campbell, G. R., Jones, W. D., Campagne, F., Wen, Z., Walker, S. J., Su, Z., Chu, T., Goodsaid, F. M., Pusztai, L., et al. (2010) · Cited by 37 · see citing works
Self-citationMultivariate Classification Rules: Calibration and Discrimination
Schmid, C. H., & Griffith, J. L. (2005) · Encyclopedia of Biostatistics · doi:10.1002/0470011815.b2a13049 · Cited by 36 · see citing works
Estimation of a convex ROC curve
Lloyd, C. J. (2002) · Statistics & Probability Letters · doi:10.1016/s0167-7152(02)00212-2 · Cited by 29 · see citing works
Horsch, K., Giger, M. L., & Metz, C. E. (2008) · Academic Radiology · doi:10.1016/j.acra.2008.04.022 · Cited by 22 · see citing works
Parametric likelihoods for multiple non-fatal competing risks and death
Shen, Y., & Thall, P. F. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980515)17:9 · Cited by 17 · see citing works
Semiparametric Estimation of the Relationship between ROC Operating Points and the Test-result Scale
Pesce, L. L., Horsch, K., Drukker, K., & Metz, C. E. (2011) · Academic Radiology · doi:10.1016/j.acra.2011.08.003 · Cited by 13 · see citing works
Using Dual Beta Distributions to Create “Proper” ROC Curves Based on Rating Category Data
Mossman, D., & Peng, H. (2015) · Medical Decision Making · doi:10.1177/0272989x15582210 · Cited by 11 · see citing works
Horsch, K., Pesce, L. L., Giger, M. L., Metz, C. E., & Jiang, Y. (2012) · Medical Physics · doi:10.1118/1.3700168 · Cited by 7 · see citing works
Investigation of methods for calibration of classifier scores to probability of disease
Chen, W., Sahiner, B., Samuelson, F. W., Pezeshk, A. X., & Petrick, N. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2082142 · Cited by 2 · see citing works
Self-citationVERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Flagged first, then by citations
2 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [21]Metz CE Herman BA Shen J Maximum-likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data Stat Med 1998 17 1033 1053 9612889
- [40]Kondratovich MV Yousef WA Evaluation of accuracy and optimal cutoff of diagnostic devices in the same study Proceedings of the 2005 joint statistical meetings, section on statistics in epidemiology 2547 2551 Alexandria, VA American Statistical Association
Cited by
Sorted by citations
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423 · see citing works
Machine Learning (ML) in Medicine: Review, Applications, and Challenges
Rahmani, A. M., Yousefpoor, E., Yousefpoor, M. S., Mehmood, Z., Haider, A., Hosseinzadeh, M., & Naqvi, R. A. (2021) · Mathematics · doi:10.3390/math9222970 · Cited by 275 · see citing works
Calibrating machine learning approaches for probability estimation: A comprehensive comparison
Ojeda, F. M., Jansen, M. L., Thiéry, A., Blankenberg, S., Weimar, C., Schmid, M., & Ziegler, A. (2023) · Statistics in Medicine · doi:10.1002/sim.9921 · Cited by 71 · see citing works
Machine learning to predict early recurrence after oesophageal cancer surgery
Rahman, S., Walker, R., Lloyd, M. A., Grace, B. L., van Boxel, G. I., Kingma, B. F., Ruurda, J. P. H., van Hillegersberg, R., Harris, S., Parsons, S. L., et al. (2020) · British journal of surgery · doi:10.1002/bjs.11461 · Cited by 56 · see citing works
Machine Learning Risk Prediction Model of 90-day Mortality After Gastrectomy for Cancer
Pera, M., Gibert, J., Gimeno, M., Garsot, E., Eizaguirre, E., Miró, M., Castro, S., Miranda, C., Reka, L., Leturio, S., et al. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005616 · Cited by 44 · see citing works
Mitigating bias in calibration error estimationPreprint
Roelofs, R., Cain, N., Shlens, J., & Mozer, M. (2020) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.2012.08668 · Cited by 158 · see citing works
Hussain, T., Nugent, C., Moore, A., Liu, J., & Beard, A. (2021) · Sensors · doi:10.3390/s21134504 · Cited by 15 · see citing works
Irvin, J., Pareek, A., Long, J., Rajpurkar, P., Eng, D. K.-M., Khandwala, N., Haug, P. J., Jephson, A. R., Conner, K. E., Gordon, B. H., et al. (2021) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000622 · Cited by 12 · see citing works
MLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 24 · see citing works
Generating diagnostic profiles of cognitive decline and dementia using magnetoencephalography
Scheijbeler, E. P., Schoonhoven, D. N., Engels, M. M. A., Scheltens, P., Stam, C. J., Gouw, A. A., & Hillebrand, A. (2021) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2021.11.002 · Cited by 10 · see citing works
Classifier Calibration: With Application to Threat Scores in Cybersecurity
Yousef, W. A., Traoré, I., & Briguglio, W. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3170011 · Cited by 9 · see citing works
Pava, M. D. L., Ríos, H. A., Rodriguez, F. J., Perdómo, O., & Gonzalez, F. A. (2021) · International Symposium on Medical Information Processing and Analysis (SIPAIM) · doi:10.1117/12.2606319 · Cited by 9 · see citing works
Kolla, L., Linn, K. A., Navathe, A. S., Kreisler, C., Roberts, C. B., Park, S.-H., Singh, H., Feng, J., Chen, J., & Parikh, R. B. (2025) · JAMA Health Forum · doi:10.1001/jamahealthforum.2025.2717 · Cited by 7 · see citing works
ShapeFlow: Dynamic Shape Interpreter for TensorFlowPreprint
Verma, S., & Su, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13452 · Cited by 7 · see citing works
Towards stimulation-free automatic electrocorticographic speech mapping in neurosurgery patients
Voskoboynikov, A., Aliverdiev, M., Nekrasova, J. Y., Semenkov, I., Skalnaya, A. A., Sinkin, M. V., & Ossadtchi, A. E. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/adfc9c · Cited by 5 · see citing works
Machine learning to predict early recurrence after oesophageal cancer surgeryPreprint
Rahman, S., Walker, R., Lloyd, M. A., Grace, B. L., van Boxel, G. I., Kingma, B. F., Ruurda, J. P. H., van Hillegersberg, R., Harris, S., Parsons, S. L., et al. (2019) · medRxiv · doi:10.1101/19001073 · Cited by 5 · see citing works
Improving the Decision Support in Diagnostic Systems Using Classifier Probability Calibration
Kortum, X., Grigull, L., Muecke, U., Lechner, W. M., & Klawonn, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03493-1_44 · Cited by 4 · see citing works
Gosztolya, G., & Tóth, L. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2552 · Cited by 3 · see citing works
Credit scoring using neural networks and SURE posterior probability calibration
Garcin, M., & Stéphan, S. (2025) · International Journal of Applied Decision Sciences · doi:10.1504/ijads.2026.151983 · Cited by 4 · see citing works
Localized Calibration: Metrics and Recalibration.Preprint
Luo, R., Bhatnagar, A., Wang, H., Xiong, C., Savarese, S., Bai, Y., Zhao, S., & Ermon, S. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
Physical and mechanical evaluation of medical devices
Ali, K. A., Shaw, T. K., Sarkar, S., Bishal, A., Mukherjee, R., & Hossain, C. M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-32828-2.00013-2 · Cited by 1 · see citing works
Classifier Calibration: with implications to threat scores in cybersecurityPreprint
Yousef, W. A., Traoré, I., & Briguglio, W. (2021) · arXiv (Cornell University) · doi:10.24433/co.4889911.v1 · Cited by 1 · see citing works
Privacy Preserving Recalibration under Domain ShiftPreprint
Luo, R., Zhao, S.-J., Song, J., Kuck, J., Ermon, S., & Savarese, S. (2020) · arXiv.org · doi:10.48550/arxiv.2008.09643 · Cited by 3 · see citing works
Mishra, D., Tiwari, A., Srivastava, S., Tripathi, M., & Kapoor, A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.23.678112
Calzone: A Python package for measuring calibration of probabilistic models for classification
Fan, K. L., Pennello, G. A., Liu, Q., Petrick, N., Samala, R. K., Samuelson, F. W., Thompson, Y. L. E., & Cao, Q. (2025) · The Journal of Open Source Software · doi:10.21105/joss.08026
Self-citationEarly Prediction of Critical Care Interventions From Pediatric Emergency Department Triage
Ha, T., Kappy, B., Chamberlain, J. M., & McKinley, K. W. (2026) · Hospital Pediatrics · doi:10.1542/hpeds.2025-009127
Zignani, N., Balzarini, M., Lopiano, G., Campagner, A., Dabizzi, E., Fracas, E., Millefanti, L., Segato, S., Cengia, G., Villanacci, V., et al. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15156094
He, S., Mehboob, S., Afghan, N., & Hung, N. M. (2026) · Journal of Food Composition and Analysis · doi:10.1016/j.jfca.2026.109474
- Authors
- Chen, W., Sahiner, B., Samuelson, F., Pezeshk, A., & Petrick, N.
- Affiliations
- Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, USA
- United States Food and Drug Administration (US)
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2016-08-08
- Volume
- 27
- Issue
- 5
- Pages
- 1394-1409
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- PMC
- PMC5548655
- OpenAlex
- W2483232919
- Cited by
- 29
- References
- 46
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 1 · 2023: 1 · 2022: 3 · 2021: 7 · 2020: 5 · 2019: 3 · 2018: 1
- FWCI
- 2.27
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion, cc-by)
- PubMed Central (acceptedVersion, cc-by)
- Funders
- National Institutes of Health (FD999999)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- disease probability, clinical decision support systems, medical diagnostic devices, uncertainty estimation, confidence intervals, parameter calibration, non-parametric calibration, discrimination performance
- MeSH terms
- Calibration, Diagnosis, Disease, Humans, Probability, Statistics as Topic, Confidence Intervals, Sample Size, Statistics, Nonparametric
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
