A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Open accessAbstract
When validating risk models (or probabilistic classifiers), calibration is often overlooked. Calibration refers to the reliability of the predicted risks, i.e. whether the predicted risks correspond to observed probabilities. In medical applications this is important because treatment decisions often rely on the estimated risk of disease. The aim of this paper is to present generic tools to assess the calibration of multiclass risk models. We describe a calibration framework based on a vector spline multinomial logistic regression model. This framework can be used to generate calibration plots and calculate the estimated calibration index (ECI) to quantify lack of calibration. We illustrate these tools in relation to risk models used to characterize ovarian tumors. The outcome of the study is the surgical stage of the tumor when relevant and the final histological outcome, which is divided into five classes: benign, borderline malignant, stage I, stage II-IV, and secondary metastatic cancer. The 5909 patients included in the study are randomly split into equally large training and test sets. We developed and tested models using the following algorithms: logistic regression, support vector machines, k nearest neighbors, random forest, naive Bayes and nearest shrunken centroids. Multiclass calibration plots are interesting as an approach to visualizing the reliability of predicted risks. The ECI is a convenient tool for comparing models, but is less informative and interpretable than calibration plots. In our case study, logistic regression and random forest showed the highest degree of calibration, and the naive Bayes the lowest.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Schölkopf, B., & Smola, A. J. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/4175.001.0001 · Cited by 11,934 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Agresti, A. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/0471249688 · Cited by 6,746 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Williams, C. K. I. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s269 · Cited by 4,390 · see citing works
Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
Golub, G. H., Heath, M. T., & Wahba, G. (1979) · Technometrics · doi:10.1080/00401706.1979.10489751 · Cited by 3,940 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
An empirical study of the naive Bayes classifier
Rish, I. (2001) · Cited by 2,532 · see citing works
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276 · see citing works
(2010) · Sage eBooks · doi:10.4135/9781412961288.n40 · Cited by 2,028 · see citing works
Rao, S. J. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s263 · Cited by 1,986 · see citing works
Probability Estimates for Multi-class Classification by Pairwise Coupling
Wu, T., Lin, C., & Weng, R. C. (2004) · Journal of Machine Learning Research · doi:10.5555/1005332.1016791 · Cited by 1,673 · see citing works
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · see citing works
Classification by pairwise coupling
Hastie, T., & Tibshirani, R. J. (1998) · The Annals of Statistics · doi:10.1214/aos/1028144844 · Cited by 1,304 · see citing works
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
Applications of Machine Learning in Cancer Prediction and Prognosis
Cruz, J. A., & Wishart, D. S. (2006) · Cancer Informatics · doi:10.1177/117693510600200030 · Cited by 1,113 · see citing works
Probability Estimates for Multi-class Classification by Pairwise Coupling
Wu, T., Lin, C., Weng, R. C., & Singer, Y. (2003) · Cited by 989 · see citing works
Applications of machine learning in cancer prediction and prognosis.
Cruz, J. A., & Wishart, D. S. (2007) · PubMed Central · Cited by 879 · see citing works
The use of receiver operating characteristic curves in biomedical informatics
Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Lucila Ohno-machado, A. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.02.008 · Cited by 821 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation and Updating
Mann, C. J. H. (2009) · Kybernetes · doi:10.1108/k.2009.06738fae.002 · Cited by 794 · see citing works
The Top Ten Algorithms in Data Mining
(2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420089653 · Cited by 682 · see citing works
Validation, updating and impact of clinical prediction rules: A review
Toll, D. B., Janssen, K. J. M., Vergouwe, Y., & Moons, K. G. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.04.008 · Cited by 584 · see citing works
Van Calster, B., Van Hoorde, K., Valentin, L., Testa, A. C., Fischerova, D., Van Holsbeke, C., Savelli, L., Franchi, D., Epstein, E., Kaijser, J., et al. (2014) · BMJ · doi:10.1136/bmj.g5920 · Cited by 510 · see citing works
Therapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507 · see citing works
Timmerman, D., Testa, A. C., Bourne, T., Ferrazzi, E. M., Ameye, L., Konstantinovic, M. L., CALSTER, B. V., Collins, W. P., Vergote, I. B., Huffel, S. V., et al. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.01.7632 · Cited by 480 · see citing works
Self-citationThe use of artificial neural networks in decision support in cancer: A systematic review
Lisboa, P. J., & Taktak, A. F. G. (2006) · Neural Networks · doi:10.1016/j.neunet.2005.10.007 · Cited by 462 · see citing works
Vector Generalized Additive Models
Yee, T. W., & Wild, C. J. (1996) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1996.tb02095.x · Cited by 430 · see citing works
Machine Learning for Detection and Diagnosis of Disease
Sajda, P. (2006) · Annual Review of Biomedical Engineering · doi:10.1146/annurev.bioeng.8.061505.095802 · Cited by 414 · see citing works
Class Prediction by Nearest Shrunken Centroids, with Applications to DNA Microarrays
Tibshirani, R. J., Hastie, T. J., Narasimhan, B., & Chu, G. (2003) · Statistical Science · doi:10.1214/ss/1056397488 · Cited by 413 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Partially parametric techniques for multiple imputation
Nathaniel Schenker, A., & Taylor, J. M. G. (1996) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(95)00057-7 · Cited by 335 · see citing works
Malley, J. D., Kruppa, J., Dasgupta, A., Malley, K. G., & Ziegler, A. (2011) · Methods of Information in Medicine · doi:10.3414/me00-01-0052 · Cited by 242 · see citing works
Calibrating predictive model estimates to support personalized medicine
Jiang, X., Osl, M., Kim, J., & Lucila Ohno-machado, A. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000291 · Cited by 209 · see citing works
Novel Artificial Neural Network for Early Detection of Prostate Cancer
Djavan, B., Mesut Remzi, A., Zlotta, A. R., Seitz, C., Snow, P. B., & Marberger, M. (2002) · Journal of Clinical Oncology · doi:10.1200/jco.2002.20.4.921 · Cited by 207 · see citing works
Timmerman, D., CALSTER, B. V., Testa, A. C., Guerriero, S., Fischerová, D., Lissoni, A. A., Holsbeke, C. V., Fruscio, R., Czekierdowski, A., Jurkovic, D., et al. (2010) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.7636 · Cited by 197 · see citing works
Self-citationAlonzo, T. A. (2009) · American Journal of Epidemiology · doi:10.1093/aje/kwp129 · Cited by 190 · see citing works
Risk estimation and risk prediction using machine-learning methods
Kruppa, J., Ziegler, A., & König, I. R. (2012) · Human Genetics · doi:10.1007/s00439-012-1194-y · Cited by 183 · see citing works
The Use of Genomics in Clinical Trial Design
Simon, R. M. (2008) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-07-4531 · Cited by 170 · see citing works
Comparison of Cox Regression With Other Methods for Determining Prediction Models and Nomograms
Kattan, M. W. (2003) · The Journal of Urology · doi:10.1097/01.ju.0000094764.56269.2d · Cited by 153 · see citing works
Practical experiences on the necessity of external validation
König, I. R., Malley, J. D., Weimar, C., Diener, H., & Ziegler, A. (2007) · Statistics in Medicine · doi:10.1002/sim.3069 · Cited by 136 · see citing works
Ensembles of nested dichotomies for multi-class problems
Frank, E., & Krämer, S. (2004) · doi:10.1145/1015330.1015363 · Cited by 121 · see citing works
Holsbeke, C. V., CALSTER, B. V., Testa, A. C., Domali, E., Lü, C., Huffel, S. V., Valentin, L., & Timmerman, D. (2009) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-08-0113 · Cited by 113 · see citing works
Self-citationPrediction of ectopic pregnancy in women with a pregnancy of unknown location
Condous, G. S., Calster, B. V., Kirk, E., et al. (2007) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.4015 · Cited by 103 · see citing works
Kruppa, J., Liu, Y., Biau, G., Kohler, M., König, I. R., Malley, J. D., & Ziegler, A. (2014) · Biometrical Journal · doi:10.1002/bimj.201300068 · Cited by 97 · see citing works
Extending the c‐statistic to nominal polytomous outcomes: the Polytomous Discrimination Index
CALSTER, B. V., Belle, V. V., Vergouwe, Y., Timmerman, D., Huffel, S. V., & Steyerberg, E. W. (2012) · Statistics in Medicine · doi:10.1002/sim.5321 · Cited by 94 · see citing works
Self-citationAssessing the discriminative ability of risk models for more than two outcome categories
CALSTER, B. V., Vergouwe, Y., Looman, C. W. N., Belle, V. V., Timmerman, D., & Steyerberg, E. W. (2012) · European Journal of Epidemiology · doi:10.1007/s10654-012-9733-3 · Cited by 92 · see citing works
Self-citationRisk prediction with machine learning and regression methods
Steyerberg, E. W., van der Ploeg, T., & CALSTER, B. V. (2014) · Biometrical Journal · doi:10.1002/bimj.201300297 · Cited by 89 · see citing works
Steinbach, M. S., & Tan, P.-N. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420089653-15 · Cited by 87 · see citing works
Discrimination and calibration of mortality risk prediction models in interventional cardiology
Matheny, M. E., Lucila Ohno-machado, A., & Resnic, F. S. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.02.007 · Cited by 77 · see citing works
Kruppa, J., Liu, Y.-F., Diener, H., Holste, T., Weimar, C., König, I., & Ziegler, A. (2014) · Biometrical journal. Biometrische Zeitschrift · doi:10.1002/bimj.201300077 · Cited by 70 · see citing works
Assessing calibration of multinomial risk prediction models
Hoorde, K. V., Vergouwe, Y., Timmerman, D., Huffel, S. V., Steyerberg, E. W., & CALSTER, B. V. (2014) · Statistics in Medicine · doi:10.1002/sim.6114 · Cited by 65 · see citing works
Self-citationThe Top Ten Algorithms in Data Mining
Rutledge, J. (2009) · Journal of Quality Technology · doi:10.1080/00224065.2009.11917798 · Cited by 62 · see citing works
Klement, R. J., Allgäuer, M., Appold, S., et al. (2014) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2013.11.216 · Cited by 58 · see citing works
COMPSTAT 1998 — Proceedings in Computational Statistics
Theus, M. (2000) · Computational Statistics · doi:10.1007/s001800000034 · Cited by 55 · see citing works
Matheny, M. E., Resnic, F. S., Arora, N., & Lucila Ohno-machado, A. (2007) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.05.008 · Cited by 36 · see citing works
Bignardi, T., Condous, G. S., Kirk, E., et al. (2010) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.7669 · Cited by 35 · see citing works
Brinkman, S., Abu–Hanna, A., van der Veen, A., et al. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e318232d7b0 · Cited by 28 · see citing works
A Framework for Evaluating Markers Used to Select Patient Treatment
Janes, H. E., PEPE, M. S., & Huang, Y. (2013) · Medical Decision Making · doi:10.1177/0272989x13493147 · Cited by 27 · see citing works
Esbroeck, A. V., Rubinfeld, I. S., Hall, B. L., et al. (2014) · Surgery · doi:10.1016/j.surg.2014.04.034 · Cited by 26 · see citing works
Jiang, X., Menon, A. K., Wang, S., Kim, J., & Lucila Ohno-machado, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0048823 · Cited by 14 · see citing works
Taktak, A. F. G., Eleuteri, A., Lake, S. P., et al. (2008) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2008.04.005 · Cited by 12 · see citing works
Clinical prognostic methods: Trends and developments
Peek, N. B., & Abu–Hanna, A. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.016 · Cited by 8 · see citing works
Shared Decision Making, Decision Aids, and Risk Communication
Helfand, M. (2007) · Medical Decision Making · doi:10.1177/0272989x070270050101 · Cited by 8 · see citing works
Vector splines and other vector smoothers
Yee, T. W. (2000) · COMPSTAT · doi:10.1007/978-3-642-57678-2_75 · Cited by 7 · see citing works
CALSTER, B. V., Luts, J., Suykens, J. A. K., Condous, G. S., Bourne, T., Timmerman, D., & Huffel, S. V. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74695-9_15 · Cited by 7 · see citing works
Self-citationHosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9
Flagged first, then by citations
Cited by
Sorted by citations
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Self-citationA calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citationValidation of prediction models in the presence of competing risks: a guide through modern methods
van Geloven, N., Giardiello, D., Bonneville, E. F., Teece, L., Ramspek, C. L., van Smeden, M., Snell, K. I. E., van Calster, B., Pohar-Perme, M., Riley, R. D., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069249 · Cited by 106 · see citing works
Self-citation
- Authors
- Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B.
- Affiliations
- Universitair Ziekenhuis Leuven (BE)
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2015-04
- Volume
- 54
- Pages
- 283-293
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2057678261
- Cited by
- 85
- References
- 70
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 13 · 2023: 3 · 2022: 6 · 2021: 11 · 2020: 10 · 2019: 11 · 2018: 3 · 2017: 6 · 2016: 2 · 2015: 1
- FWCI
- 3.57
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Lirias (submittedVersion, other-oa)
- Funders
- Agency for Innovation by Science and Technology
- Flemish Government (FWO project G049312N, IWT project TBM 070706-IOTA3, iMinds Medical Information Technologies SBO 2014)
- Research Council KU Leuven (GOA/10/09 MaNet, CoE PFV/10/002 (OPTEC)
- Belgian Federal Science Policy Office (IUAP P7/19/DYSCO)
- European Research Council (339804)
- Imperial College Healthcare NHS Trust and Imperial College London
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- probabilistic classifiers, calibration plots, ovarian neoplasms, surgical stage, histological outcomes, model validation, nearest shrunken centroids, random forest, logistic regression, naive Bayes, support vector machine, k-nearest neighbors
- MeSH terms
- Machine Learning, Adult, Aged, Algorithms, Female, Humans, Middle Aged, Ovarian Neoplasms, Models, Statistical, Logistic Models, Risk Assessment, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
