CoChat
Sign Up
PreprintCited by 1Machine Learning and Data ClassificationImbalanced Data Classification TechniquesData Mining and Machine Learning Applications

The Role of Hyperparameters in Predictive Multiplicity

Open access
Cavus, Mustafa, Woźnica, Katarzyna, Biecek, Przemysław·2025·arXiv
DOI 10.48550/arxiv.2503.13506arXiv 2503.13506
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper investigates the critical role of hyperparameters in predictive multiplicity, where different machine learning models trained on the same dataset yield divergent predictions for identical inputs. These inconsistencies can seriously impact high-stakes decisions such as credit assessments, hiring, and medical diagnoses. Focusing on six widely used models for tabular data - Elastic Net, Decision Tree, k-Nearest Neighbor, Support Vector Machine, Random Forests, and Extreme Gradient Boosting - we explore how hyperparameter tuning influences predictive multiplicity, as expressed by the distribution of prediction discrepancies across benchmark datasets. Key hyperparameters such as lambda in Elastic Net, gamma in Support Vector Machines, and alpha in Extreme Gradient Boosting play a crucial role in shaping predictive multiplicity, often compromising the stability of predictions within specific algorithms. Our experiments on 21 benchmark datasets reveal that tuning these hyperparameters leads to notable performance improvements but also increases prediction discrepancies, with Extreme Gradient Boosting exhibiting the highest discrepancy and substantial prediction instability. This highlights the trade-off between performance optimization and prediction consistency, raising concerns about the risk of arbitrary predictions. These findings provide insight into how hyperparameter optimization leads to predictive multiplicity. While predictive multiplicity allows prioritizing domain-specific objectives such as fairness and reduces reliance on a single model, it also complicates decision-making, potentially leading to arbitrary or unjustified outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.13506

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Model selection is influenced by conventional choices such as learning rate and arbitrary choices such as random seeds.

    “In addition to such intentional choices, model selection is also influenced by conventional choices, such as the selection of a learning rate, or arbitrary choices, such as the selection of random seeds (Ganesh, Taik, and Farnadi 2025;Cavus, Woźnica, and Biecek 2025).”

    From Explainable AI Isn't Enough! Rethinking Algorithmic Contestability (2026) · cites this paper as Cavus, Woźnica, and Biecek 2025

    The Introduction (p. 2) says choices of hyperparameters, including learning rates, can affect model predictions, and that randomness during training contributes to predictive multiplicity. It does not identify random-seed selection as an arbitrary choice or establish its effect on model selection (searched: random seeds, seed). (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. [1]1. Marx, C., Calmon, F., Ustun, B.: Predictive multiplicity in classification. In: ICML, pp. 6765-6774 (2020)
  2. [3]3. Roth, A., Tolbert, A., Weinstein, S.: Reconciling Individual Probability Forecasts. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 101-110 (2023)
  3. [2]2. Renard, X., Laugel, T., Detyniecki, M.: Understanding prediction discrepancies in classification. Mach. Learn. 113 (10), 7997-8026 (2024)
  4. [4]4. Du, A. Y., Ngo, D. D., Wu, Z. S.: Reconciling model multiplicity for downstream decision making. arXiv preprint arXiv:2405.19667 (2024) · arXiv
  5. [6]6. Cavus, M., Biecek, P.: Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective. arXiv preprint arXiv:2412.09712 (2024) · arXiv
  6. [5]5. Long, C., Hsu, H., Alghamdi, W., Calmon, F.: Individual arbitrariness and group fairness. Adv. Neural Inf. Process. Syst. 36 , 68602-68624 (2023)
  7. [7]7. Gomez, J.F., Machado, C., Paes, L.M., Calmon, F.: Algorithmic arbitrariness in content moderation. In: Proc. 2024 ACM Conf. Fairness, Accountability, and Transparency, pp. 2234-2253 (2024)
  8. [9]9. Cavus, M., Biecek, P.: An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification. arXiv preprint arXiv:2405.01557 (2024) · arXiv
  9. [8]8. Cooper, A.F., Lee, K., Choksi, M.Z., Barocas, S., De Sa, C., Grimmelmann, J., Kleinberg, J., Sen, S., Zhang, B.: Arbitrariness and social prediction: The confounding role of variance in fair classification. In: Proc. AAAI Conf. Artif. Intell. 38 (20), 22004-22012 (2024)
  10. [10]10. Ganesh, P., Daldaban, I.I., Cofone, I., Farnadi, G.: The cost of arbitrariness for individuals: Examining the legal and technical challenges of model multiplicity. arXiv preprint arXiv:2407.13070 (2024) · arXiv
  11. [12]12. Sokol, K., Kull, M., Chan, J., Salim, F.: Cross-model Fairness: Empirical Study of Fairness and Ethics Under Model Multiplicity. ACM J. Respons. Comput. 1 (3), 1-27 (2024)
  12. [11]11. Watson-Daniels, J., Parkes, D.C., Ustun, B.: Predictive multiplicity in probabilistic classification. In: Proc. AAAI Conf. Artif. Intell. 37 , 10306-10314 (2023)
  13. [13]13. Van Rijn, J. N., Hutter, F.: Hyperparameter importance across datasets. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2367-2376 (2018)
  14. [14]14. Probst, P., Boulesteix, A.-L., Bischl, B.: Tunability: Importance of hyperparameters of machine learning algorithms. JMLR 20 (53), 1-32 (2019)
  15. [16]16. Wickstrøm, K. K., Höhne, M., Hedström, A.: From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation. In: ECCV 2024 (2024)
  16. [15]15. Müller, S., Toborek, V., Beckh, K., Jakobs, M., Bauckhage, C., Welke, P.: An empirical evaluation of the Rashomon effect in explainable machine learning. In: ECML PKDD, pp. 462-478 (2023)
  17. [17]17. Grushetskaya, Y., Sips, M., Schachtschneider, R., Saberioon, M., Mahan, A.: HPExplorer: XAI method to explore the relationship between hyperparameters and model performance. In: ECML PKDD, pp. 319-334 (2024)
  18. [19]19. Karl, F., Pielok, T., Moosbauer, J., Pfisterer, F., Coors, S., Binder, M., Schneider, L., Thomas, J., Richter, J., Lang, M., Garrido-Merchán, E. C., Branke, J., Bischl, B.: Multi-Objective Hyperparameter Optimization in Machine Learning-An Overview. ACM Transactions on Evolutionary Learning and Optimization, 3 (4) (2023)
  19. [18]18. Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A. L., Deng, D., Lindauer, M.: Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges. arXiv preprint arXiv:2107.05847 (2021) · arXiv
  20. [20]20. Sass, R., Bergman, E., Biedenkapp, A., Hutter, F., Lindauer, M.: DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning. arXiv preprint arXiv:2206.03493 (2022). · arXiv
  21. [22]22. Wang, Q., Ming, Y., Jin, Z., Shen, Q., Liu, D., Smith, M. J., Veeramachaneni, K., Qu, H.: ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-12 (2019)
  22. [21]21. Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., Sculley, D.: Google Vizier: A service for black-box optimization. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1487-1495 (2017)
  23. [23]23. Wistuba, M., Schilling, N., Schmidt-Thieme, L.: Sequential Model-Free Hyperparameter Tuning. In: 2015 IEEE International Conference on Data Mining, pp. 1033-1038 (2015)
  24. [25]25. Pfisterer, F., van Rijn, J. N., Probst, P., Müller, A., Bischl, B.: Learning Multiple Defaults for Machine Learning Algorithms. arXiv preprint arXiv:1811.09409 (2021) · arXiv
  25. [24]24. Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., Hutter, F.: Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning. arXiv preprint arXiv:2007.04074 (2022) · arXiv
  26. [26]26. Hsu, H., Li, G., Hu, S., Chen, C.-F.: Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation. In: 12th ICLR (2024)
  27. [28]28. Biecek, P., Baniecki, H., Krzyziński, M., Cook, D.: Performance is not enough: The story told by a Rashomon Quartet. J. Comput. Graph. Stat. 33 (3), 11181121. Taylor & Francis (2024)
  28. [27]27. Breiman, L.: Statistical modeling: The two cultures. Stat. Sci. 16 (3), 199-231 (2001)
  29. [29]29. Rudin, C., Zhong, C., Semenova, L., Seltzer, M., Parr, R., Liu, J., Boner, Z.: Amazing things come from having many good models. arXiv preprint arXiv:2407.04846 (2024) · arXiv
  30. [30]30. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. B 67 (2), 301-320 (2005)
  31. [32]32. Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13 (1), 21-27 (1967)
  32. [36]36. Stando, A., Cavus, M., Biecek, P.: The effect of balancing methods on model behavior in imbalanced classification problems. In: Fifth International Workshop on Learning with Imbalanced Domains: Theory and Applications, pp. 16-30. PMLR (2024)
  33. [35]35. Chen, T., Guestrin, C.: XGBoost: A scalable tree boosting system. In: 22nd ACM SIGKDD, pp. 785-794. ACM (2016)
  34. [38]38. Mantovani, R., Horváth, T., Rossi, A. L. D., Cerri, R., Barbon Junior, S., Vanschoren, J., Carvalho, A. C. P. L. F. de: Better trees: an empirical study on hyperparameter tuning of classification decision tree induction algorithms. Data Min. Knowl. Discov. 38 (3), 1364-1416 (2024)

Cited by

Sorted by citations

Authors
Cavus, M., Woźnica, K., & Biecek, P.
ORCID
Woźnica, Katarzyna
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.13506
arXiv
2503.13506
OpenAlex
W6966896039
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
Keywords
hyperparameter optimization, prediction consistency, tabular data model, elastic net, support vector machine, extreme gradient boosting, random forest, k-nearest neighbors, decision tree, model stability, fairness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review