CoChat
Sign Up
Cited by 3Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Artificial Intelligence in Law

Perceptions of the Fairness Impacts of Multiplicity in Machine Learning

Open access
Meyer, Anna P., Kim, Yea-Seul, D'Antoni, Loris, Albarghouthi, Aws·2025·Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3706598.3713524
Open this paper in CoChat — free account

Abstract

Machine learning (ML) is increasingly used in high-stakes settings, yet multiplicity - the existence of multiple good models - means that some predictions are essentially arbitrary. ML researchers and philosophers posit that multiplicity poses a fairness risk, but no studies have investigated whether stakeholders agree. In this work, we conduct a survey to see how multiplicity impacts lay stakeholders' - i.e., decision subjects' - perceptions of ML fairness, and which approaches to address multiplicity they prefer. We investigate how these perceptions are modulated by task characteristics (e.g., stakes and uncertainty). Survey respondents think that multiplicity threatens the fairness of model outcomes, but not the appropriateness of using the model, even though existing work suggests the opposite. Participants are strongly against resolving multiplicity by using a single model (effectively ignoring multiplicity) or by randomizing the outcomes. Our results indicate that model developers should be intentional about dealing with multiplicity in order to maintain fairness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3706598.3713524

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • PWS participants showed strong intuition on which label 'best' captures the speech patterns, but they were also conscious about the space of interpretation and the fairness risk when deterministically assigning one label as the 'ground truth'.

    “While PWS participants showed strong intuition on which label 'best' captures the speech patterns, they were also conscious about the space of interpretation and the fairness risk when deterministically assigning one label as the 'ground truth' [69].”

    From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [69]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. [3][3] Peter L Bartlett and Marten H Wegkamp. 2008. Classification with a Reject Option using a Hinge Loss. Journal of Machine Learning Research 9, 8 (2008), 1823-1840. -
  2. [7][7] Emily Black, Klas Leino, and Matt Fredrikson. 2022. Selective Ensembles for Consistent Predictions. In International Conference on Learning Representations . https://openreview.net/forum?id=HfUyCRBeQc -
  3. [12][12] John Broome. 1990. Fairness. In Proceedings of the Aristotelian society , Vol. 91. JSTOR, [Aristotelian Society, Wiley], 87-101. -
  4. [17][17] Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81) , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 134-148. https://proceedings.mlr.press/ v81/chouldechova18a.html -
  5. [18][18] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019. Certified Adversarial Robustness via Randomized Smoothing. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 1310-1320. https://proceedings.mlr.press/v97/cohen19c.html -
  6. [20][20] Jason A Colquitt and Jessica B Rodell. 2015. Measuring justice and fairness. The Oxford Handbook of Justice in the Workplace 1 (2015), 187-202. -
  7. [24][24] Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2022. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research 23, 226 (2022), 1-61. -
  8. [29][29] Lisa A Elkin, Matthew Kay, James J Higgins, and Jacob O Wobbrock. 2021. An aligned rank transform procedure for multifactor contrast tests. In The 34th annual ACM symposium on user interface software and technology . 754-768. -
  9. [33][33] John Fox and Sanford Weisberg. 2019. An R Companion to Applied Regression (third ed.). Sage, Thousand Oaks CA. https://socialsciences.mcmaster. ca/jfox/Books/Companion/ -
  10. [37][37] Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems 29 (2016). -
  11. [41][41] Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics 6, 2 (1979), 65-70. -
  12. [43][43] Shomik Jain, Kathleen Creel, and Ashia Camage Wilson. 2024. Position: Scarce Resource Allocations That Rely On Machine Learning Should Be Randomized. In Forty-first International Conference on Machine Learning . https://openreview.net/forum?id=44qxX6Ty6F -
  13. [52][52] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. Advances in Neural Information Processing Systems 30 (2017). -
  14. [54][54] Cassidy Laidlaw and Soheil Feizi. 2019. Playing it safe: Adversarial robustness with an abstain option. arXiv preprint arXiv:1911.11253 (2019). - · arXiv
  15. [64][64] Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive Multiplicity in Classification. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119) , Hal Daumé III and Aarti Singh (Eds.). PMLR, 6765-6774. https: //proceedings.mlr.press/v119/marx20a.html -
  16. [66][66] Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Conference on Fairness, accountability and transparency (Proceedings of Machine Learning Research, Vol. 81) , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, PMLR, 107-118. https://proceedings.mlr.press/v81/menon18a.html -
  17. [75][75] Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner. 2024. Position: Amazing Things Come From Having Many Good Models. In Forty-first International Conference on Machine Learning . https://openreview.net/forum?id=oFDFGd9Age Manuscript submitted to ACM -
  18. [84][84] Ayelet Sela. 2018. Can computers be fair: How automated and human-powered online dispute resolution affect procedural justice in mediation and arbitration. Ohio St. J. on Disp. Resol. 33 (2018), 91. -
  19. [89][89] Joseph P Simmons, Leif D Nelson, and Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science 22, 11 (2011), 1359-1366. -
  20. [100][100] Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. 2022. Exploring the Whole Rashomon Set of Sparse Decision Trees. In Advances in Neural Information Processing Systems , Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.). Curran Associates, Inc., 14071-14084. https://openreview.net/forum?id=WHqVVk3UHr -

Cited by

Sorted by citations

Related

Authors
Meyer, A. P., Kim, Y.-S., D'Antoni, L., & Albarghouthi, A.
Affiliations
  • University of Wisconsin–Madison (US)
  • Apple (United States) (US)
  • University of California - San Diego, San Diego, California, USA
ORCID
Venue
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2025-04-25
Pages
1-15
DOI
10.1145/3706598.3713524
OpenAlex
W4409749612
Cited by
1
References
80
Citations by year
2026: 1
FWCI
1.02
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
NSF (National Science Foundation) (1918211,2446711)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
Keywords
algorithmic fairness, public perception, algorithmic decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review