Understanding Decision Subjects' Fairness Perceptions and Retention in Repeated Interactions with AI-Based Decision Systems
Open accessAbstract
The wide application of AI-based decision systems in many high-stake domains has raised concerns regarding fairness of these systems. As these systems will lead to real-life consequences to people who are subject to their decisions, understanding what these decision subjects perceive as a fair or unfair system is of vital importance. In this paper, we extend prior work in this direction by taking a perspective of repeated interactions---We ask that when decision subjects interact with an AI-based decision system repeatedly and can strategically respond to the system by determining whether to stay in the system, what factors will affect the decision subjects' fairness perceptions and retention in the system and how. To answer these questions, we conducted two randomized human-subject experiments in the context of an AI-based loan lending system. Our results suggest that in repeated interactions with the AI-based decision system, overall, decision subjects' fairness perceptions and retention in the system are significantly affected by whether the system is in favor of the group that subjects themselves belong to, rather than whether the system treats different groups in an unbiased way. However, decision subjects with different qualification levels have different reactions to the AI system's biased treatment across groups or the AI system's tendency to favor/disfavor their own group. Finally, we also find that while subjects' retention in the AI-based decision system is largely driven by their own prospects of receiving the favorable decision from the system, their fairness perceptions of the system is influenced by the system's treatment to people in other groups in a complex way.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Under repeated interactions with algorithmic decision-making systems, decision subjects' fairness perceptions are affected by the favorability of the system towards the group that the decision subjects belong to.
“Previous research has shown that under repeated interactions with algorithmic decisionmaking systems, decision subjects' fairness perceptions are afected by the favorability of the system towards the group that the decision subjects belong to [37].”
From Disentangling Fairness Perceptions in Algorithmic Decision-Making: the Effects of Explanations, Human Oversight, and Contestability (2023) · cites this paper as [37]
Previous research has shown that, under repeated interactions, system favorability towards the group that the decision subject belongs to has an effect on fairness perceptions.
“Previous research has shown that, under repeated interactions, system favorability towards the group that the decision subject belongs to has an efect on fairness perceptions [37].”
From Disentangling Fairness Perceptions in Algorithmic Decision-Making: the Effects of Explanations, Human Oversight, and Contestability (2023) · cites this paper as [37]
Certain AI-driven credit scoring models have exhibited potential bias towards specific demographic groups.
“Certain AI-driven credit scoring models have exhibited potential bias towards specific demographic groups [288,289].”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [288]
AI fairness is applied with bias migration strategies to counteract bias in credit-scoring models and ensure equitable access to loans and credit opportunities for all individuals.
“To counteract bias in credit-scoring models and ensure equitable access to loans and credit opportunities for all individuals, AI fairness is applied with bias migration strategies [287,288].”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [288]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · see citing works
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040 · see citing works
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 1× · see citing works
“Other key influencing factors include whether and how the system’s decisions are explained [3, 10, 38], the ways that the AI system’s decisions are presented and visualized [41], and people’s personal experience related to the algorithmic decision making scenario [14].”
BackgroundAlgorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · see citing works
Self-citationAttachment and trust in artificial intelligence
Gillath, O., Ai, T., Branicky, M. S., Keshmiri, S., Davison, R. B., & Spaulding, R. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106607 · Cited by 448 · cited 1× · see citing works
“There is a growing body of work on understanding how humans adopt, interact with, and trust the AI-based decision-making systems [7, 9, 13, 26, 29, 34–36, 45, 47, 52].”
BackgroundExplaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286 · see citing works
Affective Processes in Human–Automation Interactions
Merritt, S. M. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811411912 · Cited by 260 · see citing works
Adaptive trust calibration for human-AI collaboration
Okamura, K., & Yamada, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229132 · Cited by 239 · see citing works
Wang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211 · see citing works
How Do Fairness Definitions Fare?
Saxena, N. A., Huang, K., DeFilippis, E., Radanović, G., Parkes, D. C., & Liu, Y. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314248 · Cited by 202 · see citing works
Millions of black people affected by racial bias in health-care algorithms
Ledford, H. (2019) · Nature · doi:10.1038/d41586-019-03228-6 · Cited by 189 · see citing works
Srivastava, M., Heidari, H., & Krause, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330664 · Cited by 179 · see citing works
D’Amour, A., Srinivasan, H., Atwood, J. R., Baljekar, P. N., Sculley, D., & Halpern, Y. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372878 · Cited by 156 · see citing works
On Formalizing Fairness in Prediction with Machine LearningPreprint
Gajane, P., & Pechenizkiy, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03184 · Cited by 152 · see citing works
Risk Attitudes and Political Participation
Kam, C. D. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2012.00605.x · Cited by 141 · see citing works
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · see citing works
Self-citationAn empirical study on the perceived fairness of realistic, imperfect machine learning models
Harrison, G., Hanson, J., Jacinto, C., Ramirez, J., & Ur, B. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372831 · Cited by 114 · see citing works
Multi–objective Evolutionary Algorithms for the Risk–return Trade–off in Bank Loan Management
Mukerjee, A., Biswas, R., Deb, K., et al. (2002) · International Transactions in Operational Research · doi:10.1111/1475-3995.00375 · Cited by 91 · see citing works
Effect of Information Presentation on Fairness Perceptions of Machine Learning Predictors
van Berkel, N., Goncalves, J., Russo, D., Hosio, S., & Skov, M. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445365 · Cited by 84 · see citing works
Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems
Cheng, H.-F., Stapleton, L., Wang, R., Bullock, P., Chouldechova, A., Wu, Z. S. S., & Zhu, H. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445308 · Cited by 64 · see citing works
I agree with the decision, but they didn't deserve this
Kasinidou, M., Kleanthous, S., Barlas, P., & Otterbacher, J. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445931 · Cited by 61 · see citing works
Shen, H., Jin, H., Cabrera, Á. A., Perer, A., Zhu, H., & Hong, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415224 · Cited by 57 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2011) · arXiv · doi:10.48550/arxiv.1104.3913 · Cited by 55 · see citing works
You’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · see citing works
Self-citationWill You Accept the AI Recommendation? Predicting Human Behavior in AI-Assisted Decision Making
Wang, X., Lu, Z., & Yin, M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512240 · Cited by 49 · see citing works
Self-citationThe disparate equilibria of algorithmic decision making when individuals invest rationally
Liu, L. T., Wilson, A., Haghtalab, N., Kalai, A. T., Borgs, C., & Chayes, J. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372861 · Cited by 47 · see citing works
A Study on Fairness and Trust Perceptions in Automated Decision MakingPreprint
Schoeffer, J., Machowski, Y., & Kuehl, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04757 · Cited by 31 · see citing works
A Human-in-the-loop Framework to Construct Context-aware Mathematical Notions of Outcome Fairness
Yaghini, M., Krause, A., & Heidari, H. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462583 · Cited by 21 · see citing works
How Do Fair Decisions Fare in Long-term Qualification?Preprint
Zhang, X., Tu, R., Liu, Y., Liu, M., Kjellström, H., Zhang, K., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11300 · Cited by 17 · see citing works
Who Gets What, According to Whom? An Analysis of Fairness Perceptions in Service Allocation
Hannan, J., Chen, H.-Y. W., & Joseph, K. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462568 · Cited by 14 · see citing works
Precarity: Modeling the Long Term Effects of Compounded Decisions on Individual Instability
Nokhiz, P., Ruwanpathirana, A. K., Patwari, N., & Venkatasubramanian, S. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462529 · Cited by 7 · see citing works
Dimensions of Diversity in Human Perceptions of Algorithmic FairnessPreprint
Grgić-Hlača, N., Lima, G., Weller, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00808 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In International Conference on Machine Learning. PMLR, 60-69.
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu- racy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77-91.
- Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016. Equality of Op- portunity in Supervised Learning. In Advances in Neural Information Process- ing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2016/file/ 9d2682367c3935defcb1f9e247a97c0d-Paper.pdf
- Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and information systems 33, 1 (2012), 1-33.
- Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed impact of fair machine learning. In International Conference on Machine Learning. PMLR, 3150-3158.
- Karima Makhlouf, Sami Zhioua, and Catuscia Palamidessi. 2020. On the applica- bility of ml fairness notions. arXiv preprint arXiv:2006.16745 (2020). · arXiv
- Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A Survey on Bias and Fairness in Machine Learning. CoRR abs/1908.09635 (2019). arXiv:1908.09635 http://arxiv.org/abs/1908.09635 · arXiv
- Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447-453.
- Amy Rechkemmer and Ming Yin. 2022. When Confidence Meets Accuracy: Exploring the Effects of Multiple Performance Indicators on Trust in Machine Learning Models. In CHI Conference on Human Factors in Computing Systems. 1-14.
- Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International conference on machine learning. PMLR, 325-333.
- Xueru Zhang, Mohammadmahdi Khaliligarekani, Cem Tekin, et al. 2019. Group retention when using machine learning in sequential decision making: the in- terplay between user dynamics and fairness. Advances in Neural Information Processing Systems 32 (2019).
Cited by
Sorted by citations
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155 · see citing works
Yurrita, M., Draws, T., Balayn, A., Murray-Rust, D., Tintarev, N., & Bozzon, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581161 · Cited by 73 · see citing works
“Previous research has shown that under repeated interactions with algorithmic decisionmaking systems, decision subjects’ fairness perceptions are afected by the favorability of the system towards the group that the decision subjects belong to [37].”
Background
- Authors
- Gemalmaz, M. A., & Yin, M.
- Affiliations
- Purdue University West Lafayette (US)
- ORCID
- Yin, Ming
- Venue
- Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2022-07-26
- Pages
- 295-306
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W4288058254
- Cited by
- 10
- References
- 39
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 4 · 2022: 1
- FWCI
- 1.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion, cc-by)
- Funders
- NSF Program on Fairness in Artificial Intelligence in Collaboration with Amazon (IIS-2040800)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Experimental Behavioral Economics Studies
- Keywords
- algorithmic fairness, perceived fairness, user retention, repeated interaction, group bias, strategic responses, algorithmic discrimination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
