Measuring the Stability of Results From Supervised Statistical Learning
The DOI matches its Crossref record; 50 of the 58 references it cites were found in a registry or an index.
Abstract
Stability is a major requirement to draw reliable conclusions when interpreting results from supervised statistical learning. In this article, we present a general framework for assessing and comparing the stability of results, which can be used in real-world statistical learning applications as well as in simulation and benchmark studies. We use the framework to show that stability is a property of both the algorithm and the data-generating process. In particular, we demonstrate that unstable algorithms (such as recursive partitioning) can produce stable results when the functional form of the relationship between the predictors and the response matches the algorithm. Typical uses of the framework in practical data analysis would be to compare the stability of results generated by different candidate algorithms for a dataset at hand or to assess the stability of algorithms in a benchmark study. Code to perform the stability analyses is provided in the form of an R package. Supplementary material for this article is available online.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research in this area is for example addressing stability [89], fairness [90], and privacy [91].”
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“…regression with (cid:7) 1 penalty (Friedman, Hastie, and Tibshirani 2010), conditional inference trees (Hothorn, Hornik, and Zeileis 2006), tree-based gradient boosting (Friedman 2001), random forests (Breiman 2001a), and support vector machines (Vapnik 1998) with a radial basis function kernel.”
MethodWu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statistics and Computing · doi:10.1007/978-0-387-21706-2 · Cited by 24,305
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Classification and Regression by randomForest
Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 4×
“We have used the total variation distance (TVD) to compute similarity and the Kappa statistic (see, e.g., Kuhn and Johnson 2013, chap.”
MethodInfluential
Research works printed but not linked: 7
- Bousquet O., 2002, Journal of Machine Learning Research, 2, 499
- Breiman L., 1984, Classification and Regression Trees (Wadsworth Statistics/Probability Series)
- Frank E., 2016, The WEKA Workbench. Online Appendix for “Data Mining: Practical Machine Learning Tools and Techniques”
- Hothorn T., 2015, Journal of Machine Learning Research, 16, 3905
- Lange T., 2002, Advances in Neural Information Processing Systems, 15, 617
- Liaw A., 2002, R News, 2, 18
- Philipp M., 2016, Proceedings of COMPSTAT 2016–22nd International Conference on Computational Statistics, 315
Cited by
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211
“Some steps have been made towards quantifying model complexity, such as using functional decomposition and quantifying the complexity of the components [71] or measuring the stability of predictions [79].”
BackgroundQuantifying Model Complexity via Functional Decomposition for Better Post-hoc Interpretability
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-43823-4_17 · Cited by 64
Interpretable machine learning for psychological research: Opportunities and pitfalls.
Henninger, M., Debelak, R., Rothacher, Y., & Strobl, C. (2023) · Psychological Methods · doi:10.1037/met0000560 · Cited by 74
Self-citation
Metadata
- Authors
- Philipp, M., Rusch, T., Hornik, K., & Strobl, C.
- Affiliations
- University of Zurich (CH)
- Vienna University of Economics and Business (AT)
- Venue
- Journal of Computational and Graphical Statistics
- Publisher
- Informa UK Limited
- Published
- 2018-08-24
- Volume
- 27
- Issue
- 4
- Pages
- 685-700
- ISSN
- 1061-8600 (ISSN-L)
- 1537-2715
- OpenAlex
- W2589781122
- Cited by
- 56
- References
- 58
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 9 · 2023: 8 · 2022: 16 · 2021: 9 · 2020: 5 · 2019: 2
- FWCI
- 3.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Computational and Graphical Statistics (publishedVersion)
- WU Research (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- recursive partitioning, algorithm stability, data generating process, benchmark study, algorithm comparison, R package
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring the Stability of Results From Supervised Statistical Learning. https://app.cochat.ai/papers/10.1080/10618600.2018.1473779
- The paper
- Philipp, M., Rusch, T., Hornik, K., & Strobl, C. (2018). Measuring the Stability of Results From Supervised Statistical Learning. Journal of Computational and Graphical Statistics. https://doi.org/10.1080/10618600.2018.1473779
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.