An empirical study on the perceived fairness of realistic, imperfect machine learning models
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
There are many competing definitions of what statistical properties make a machine learning model fair. Unfortunately, research has shown that some key properties are mutually exclusive. Realistic models are thus necessarily imperfect, choosing one side of a trade-off or the other. To gauge perceptions of the fairness of such realistic, imperfect models, we conducted a between-subjects experiment with 502 Mechanical Turk workers. Each participant compared two models for deciding whether to grant bail to criminal defendants. The first model equalized one potentially desirable model property, with the other property varying across racial groups. The second model did the opposite. We tested pairwise trade-offs between the following four properties: accuracy; false positive rate; outcomes; and the consideration of race. We also varied which racial group the model disadvantaged. We observed a preference among participants for equalizing the false positive rate between groups over equalizing accuracy. Nonetheless, no preferences were overwhelming, and both sides of each trade-off we tested were strongly preferred by a non-trivial fraction of participants. We observed nuanced distinctions between participants considering a model "unbiased" and considering it "fair." Furthermore, even when a model within a trade-off pair was seen as fair and unbiased by a majority of participants, we did not observe consensus that a machine learning model was preferable to a human judge. Our results highlight challenges for building machine learning models that are perceived as fair and broadly acceptable in realistic situations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 3×
“In contrast, we chose not to test more complex definitions like equalized odds, which requires equalizing both the true and false positive rates [12].”
BackgroundMethodInfluentialFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 5×
“Unfortunately, some key definitions of fairness are incompatible [5, 17].”
BackgroundInfluentialAwad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859 · cited 1×
“Awad et al. use a viral game to study variations across cultures in how participants would solve a variation of the trolley problem involving different demographic characteristics of the parties [3].”
BackgroundFairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · cited 1×
“A large literature proposes techniques for auditing models for such correlations [1, 6, 21, 23].”
BackgroundZafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · cited 1×
“Furthermore, the use of race may be necessary during model development to audit the model or to achieve group fairness [29].”
BackgroundAvoiding Discrimination through Causal Reasoning
Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., & Schölkopf, B. (2017) · arXiv · doi:10.48550/arxiv.1706.02744 · Cited by 358
Auditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280 · cited 1×
“A large literature proposes techniques for auditing models for such correlations [1, 6, 21, 23].”
BackgroundA Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256
Efficient Search for Diverse Coherent Explanations
Russell, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287569 · Cited by 218 · cited 1×
“A large literature proposes techniques for auditing models for such correlations [1, 6, 21, 23].”
BackgroundGrgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Srivastava, M., Heidari, H., & Krause, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330664 · Cited by 179 · cited 3×
“Srivastava et al. found a preference for demographic parity (equalizing the percentage of people who receive a positive classification, which was our = Outcome condition) over other definitions of fairness, like equalized false positive rates or false negative rates [26].”
BackgroundGrgić-Hlača, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · arXiv · doi:10.48550/arxiv.1802.09548 · Cited by 75
Cited by
Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316
“The results indicated that respondents favored an algorithm that equalizes the false positive rate between groups over one that equalizes accuracy (Harrison et al., 2020).”
BackgroundInfluentialTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
Fairness and Explanation in AI-Informed Decision Making
Angerschmid, A., Zhou, J., Theuermann, K., Chen, F., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4020026 · Cited by 161
The Moral Psychology of Artificial Intelligence
Bonnefon, J.-F., Rahwan, I., & Shariff, A. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-030123-113559 · Cited by 151
Metadata
- Authors
- Harrison, G., Hanson, J., Jacinto, C., Ramirez, J., & Ur, B.
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 392-402
- OpenAlex
- W3014011525
- Cited by
- 110
- References
- 15
- FWCI
- 17.23
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.