Effect of Information Presentation on Fairness Perceptions of Machine Learning Predictors
Open accessAbstract
The uptake of artificial intelligence-based applications raises concerns about the fairness and transparency of AI behaviour. Consequently, the Computer Science community calls for the involvement of the general public in the design and evaluation of AI systems. Assessing the fairness of individual predictors is an essential step in the development of equitable algorithms. In this study, we evaluate the effect of two common visualisation techniques (text-based and scatterplot) and the display of the outcome information (i.e., ground-truth) on the perceived fairness of predictors. Our results from an online crowdsourcing study (N = 80) show that the chosen visualisation technique significantly alters people’s fairness perception and that the presented scenario, as well as the participant’s gender and past education, influence perceived fairness. Based on these results we draw recommendations for future work that seeks to involve non-experts in AI fairness evaluations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval, recidivism prediction, house price estimation, and student admission.
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [124]
The Introduction identifies AI decision-support applications in loan approval and describes the COMPAS recidivism risk prediction system; the Method section also uses lending and recidivism scenarios. The paper does not establish that people tend to rely on AI advice for these tasks because of inherent uncertainty. House-price estimation and student admission have separate citation groups in the citing sentence and are not attributed to this source. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615 · see citing works
Herne, H., Cooley, W. W., & Lohnes, P. R. (1973) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344428 · Cited by 35,801 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Generalized linear mixed models: a practical guide for ecology and evolution
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2008.10.008 · Cited by 8,910 · see citing works
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
ggeffects: Tidy Data Frames of Marginal Effects from Regression Models
Lüdecke, D. (2018) · The Journal of Open Source Software · doi:10.21105/joss.00772 · Cited by 2,742 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · see citing works
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941 · see citing works
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859 · see citing works
Which is the fair sex? Gender differences in altruism.
Andreoni, J., & Vesterlund, L. (2001) · The Quarterly Journal of Economics · doi:10.1162/003355301556419 · Cited by 1,427 · see citing works
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · see citing works
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · see citing works
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 2× · see citing works
“A case-based explanation, in which a representative example from the training data is presented, resulted in the most consistent negative impact on participants’ justice perceptions [7].”
BackgroundKaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
<i>Why and why not</i> explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · see citing works
The What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., WATTENBERG, M. P., Viégas, F. B., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 512 · see citing works
Jagtiani, J. A., & Lemieux, C. (2019) · Financial Management · doi:10.1111/fima.12295 · Cited by 433 · see citing works
Risk assessment in social lending via random forests
Malekipirbazari, M., & Aksakallı, V. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.02.001 · Cited by 398 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 4× · see citing works
“highlights that white-box explanations do not increase participants self-reported level of understanding as compared to a black-box explanation [11] (although their objective understanding does increase), which the authors attribute to increased complexity and required cognitive load.”
BackgroundInfluentialHohman, F., Head, A., Caruana, R., DeLine, R. A., & Drucker, S. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225 · see citing works
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220 · see citing works
Wang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211 · see citing works
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207 · see citing works
How Do Fairness Definitions Fare?
Saxena, N. A., Huang, K., DeFilippis, E., Radanović, G., Parkes, D. C., & Liu, Y. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314248 · Cited by 202 · see citing works
Tait, A. R., Voepel‐Lewis, T., Zikmund‐Fisher, B. J., & Fagerlin, A. (2010) · Journal of Health Communication · doi:10.1080/10810730.2010.492560 · Cited by 199 · see citing works
Fairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199 · cited 1× · see citing works
“The defnitions included are ‘individual fairness’ [15], ‘never favour a worse individual over a better one’ [32], and ‘calibrated fairness’ [39].”
BackgroundA Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 190 · see citing works
Srivastava, M., Heidari, H., & Krause, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330664 · Cited by 179 · see citing works
Variables on Scatterplots Look More Highly Correlated When the Scales Are Increased
Cleveland, W. S., Diaconis, P. W., & McGill, R. (1982) · Science · doi:10.1126/science.216.4550.1138 · Cited by 177 · see citing works
Ethical Dimensions of Visualization Research
Correll, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300418 · Cited by 150 · see citing works
Automated Decision Making Comes of Age
Davenport, T. H., & Harris, J. G. (2005) · MIT Sloan Management Review · Cited by 121 · see citing works
Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113 · see citing works
Comparing Effectiveness and Engagement of Data Comics and Infographics
Wang, Z., Wang, S.-H., Farinella, M., Murray-Rust, D., Riche, N. H., & Bach, B. (2019) · International Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300483 · Cited by 98 · see citing works
Raub, M. (2018) · Arkansas Law Review · doi:10.54119/alr.npth2166 · Cited by 96 · see citing works
Using the Short Graph Literacy Scale to Predict Precursors of Health Behavior Change
Okan, Y., Janssen, E., Galešić, M., & Waters, E. A. (2019) · Medical Decision Making · doi:10.1177/0272989x19829728 · Cited by 90 · see citing works
Gonçalves, J., Ferreira, D., Hosio, S. J., Yong, L., Rogstadius, J., Kukka, H., & Kostakos, V. (2013) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2493432.2493481 · Cited by 88 · see citing works
Self-citationHuman-centred artificial intelligence: a contextual morality perspective
van Berkel, N., Tag, B., Gonçalves, J., & Hosio, S. J. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1818828 · Cited by 83 · see citing works
Self-citationCrowdsourcing Perceptions of Fair Predictors for Machine Learning
van Berkel, N., Gonçalves, J., Hettiachchi, D., Wijenayake, S., Kelly, R., & Kostakos, V. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359130 · Cited by 73 · see citing works
The impact of social information on visual judgments
Hullman, J. R., Adar, E., & Shah, P. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979157 · Cited by 72 · see citing works
Epstein, Z., Pennycook, G., & Rand, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376232 · Cited by 67 · cited 1× · see citing works
“Both the HCI and the AI literature highlight an increasing efort and interest towards the active involvement of the general public and/or end-users in data collection [27], training [17], and the evaluation [60] of AI applications.”
BackgroundMore Text Please! Understanding and Supporting the Use of Visualization for Clinical Text Overview
Sultanum, N., Brudno, M., Wigdor, D., & Chevalier, F. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173996 · Cited by 57 · see citing works
Shen, H., Jin, H., Cabrera, Á. A., Perer, A., Zhu, H., & Hong, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415224 · Cited by 57 · see citing works
Gender trends in computer science authorship
Wang, L. L., Stanovsky, G., Weihs, L., & Etzioni, O. (2021) · Communications of the ACM · doi:10.1145/3430803 · Cited by 55 · see citing works
Trends and Applications of Machine Learning in Quantitative FinancePreprint
Emerson, S., Kennedy, R., O'Shea, L., & O’Brien, J. J. (2019) · SSRN Electronic Journal · Cited by 52 · see citing works
Are you sure your software is gender-neutral?
Williams, G. (2014) · interactions · doi:10.1145/2524808 · Cited by 51 · see citing works
van Berkel, N., Papachristos, E., Giachanou, A., Hosio, S., & Skov, M. B. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420106 · Cited by 51 · see citing works
Self-citationYu, B., Yuan, Y., Terveen, L., Wu, Z. S., Forlizzi, J., & Zhu, H. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395528 · Cited by 49 · see citing works
An Empirical Study on the Reliability of Perceiving Correlation Indices using Scatterplots
Sher, V., Bemis, K. G., Liccardi, I., & Chen, M. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13168 · Cited by 31 · see citing works
Using Animation to Alleviate Overdraw in Multiclass Scatterplot Matrices
Chen, H. W., Engle, S. J., Joshi, A. P., Ragan, E. D., Yuksel, B. F., & Harrison, L. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173991 · Cited by 31 · see citing works
Artificial Intelligence and Policing: First QuestionsPreprint
Joh, E. E. (2018) · SSRN Electronic Journal · Cited by 29 · see citing works
Demographics and discussion influence views on algorithmic fairnessPreprint
Pierson, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09124 · Cited by 26 · see citing works
Communicating quantitative information about unexploded ordnance risks to the public.
Gibson, J., Rowe, A., Stone, E. R., & de Bruin, W. B. (2013) · Environmental Science and Technology · doi:10.1021/es305254j · Cited by 22 · see citing works
Fueling AI with public displays?
Hosio, S. J., Alorwu, A., van Berkel, N., López, M. B., Seetharaman, M., Oppenlaender, J., & Gonçalves, J. (2019) · ACM International Symposium on Pervasive Displays (PerDis) · doi:10.1145/3321335.3324943 · Cited by 8 · see citing works
Self-citationArtificial Intellegence and Policing: First Questions
Joh, E. E. (2018) · bepress Legal Repository · Cited by 5 · see citing works
Flagged first, then by citations
1 of the 61 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C. Parkes. 2017. Calibrated Fairness in Bandits. arXiv:cs.LG/1707.01875
- Emma Pierson. 2017. Demographics and discussion infuence views on algorith- mic fairness. arXiv:cs.CY/1712.09124
Cited by
Sorted by citations
Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316 · see citing works
A survey on datasets for fairness‐aware machine learning
Le Quy, T., Roy, A., Iosifidis, V., Zhang, W., & Ntoutsi, E. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1452 · Cited by 261 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
Jang, Y., Choi, S., & Kim, H. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11086-5 · Cited by 150 · see citing works
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · see citing works
Forever focused on fairness: 75 years of organizational justice in Personnel Psychology
Colquitt, J. A., Hill, E. T., & David De Cremer, A. (2022) · Personnel Psychology · doi:10.1111/peps.12556 · Cited by 97 · see citing works
Fairness Perceptions of Artificial Intelligence: A Review and Path Forward
Narayanan, D., Nagpal, M., McGuire, J., Schweitzer, S., & Cremer, D. D. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2210890 · Cited by 87 · see citing works
Chai, F., Ma, J., Wang, Y., Zhu, J., & Han, T. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1221177 · Cited by 91 · see citing works
Yurrita, M., Draws, T., Balayn, A., Murray-Rust, D., Tintarev, N., & Bozzon, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581161 · Cited by 73 · see citing works
Tal, A. S., Kuflik, T., & Kliger, D. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2095705 · Cited by 69 · see citing works
A Critical Survey on Fairness Benefits of Explainable AI
Deck, L., Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658990 · Cited by 86 · see citing works
Mitigating knowledge imbalance in AI-advised decision-making through collaborative user involvement
Gómez, C., Unberath, M., & Huang, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102977 · Cited by 55 · see citing works
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42 · see citing works
Towards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533118 · Cited by 42 · see citing works
"I'm not sure what difference is between their content and mine, other than the person itself"
Ma, R., & Kou, Y. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555150 · Cited by 45 · see citing works
van Berkel, N., Sarsenbayeva, Z., & Gonçalves, J. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102954 · Cited by 41 · see citing works
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · see citing works
“Recent studies have explored variables that contribute to the fairness [31, 76, 124, 130] and trustworthiness [34, 48, 80, 139] of AI systems, as well as the impact of assigning different decision-making roles to humans and AI on the reliance on such systems [52, 103, 122, 144].”
BackgroundHe, G., Buijsman, S., & Gadiraju, U. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3610067 · Cited by 59 · see citing works
van Berkel, N., Bellio, M., Skov, M. B., & BLANDFORD, A. E. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3571815 · Cited by 37 · see citing works
Self-citationWho Do We Mean When We Talk About Visualization Novices?
Burns, A., Lee, C., Chawla, R., Peck, E. M., & Mahyar, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581524 · Cited by 37 · see citing works
Ghasemaghaei, M., & Kordzadeh, N. (2024) · Information Systems Journal · doi:10.1111/isj.12572 · Cited by 35 · see citing works
Do Humans Prefer Debiased AI Algorithms? A Case Study in Career Recommendation
Wang, C., Wang, K., Bian, A., Islam, R., Keya, K. N., Foulds, J. R., & Pan, S. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511108 · Cited by 28 · see citing works
Differential Fairness: An Intersectional Framework for Fair AI
Islam, R., Keya, K. N., Pan, S., Sarwate, A. D., & Foulds, J. R. (2023) · Entropy · doi:10.3390/e25040660 · Cited by 27 · see citing works
Tal, A. S., Kuflik, T., Kliger, D., & Mancini, A. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2348843 · Cited by 26 · see citing works
From anecdote to evidence: the relationship between personality and need for cognition of developers
Russo, D., Masegosa, A. R., & Stol, K.-J. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10106-1 · Cited by 25 · see citing works
Self-citationHaque, M. R., Saxena, D., Weathington, K., Chudzik, J., & Guha, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642738 · Cited by 25 · see citing works
Ebrahimi, S., Abdelhalim, E., Hassanein, K., & Head, M. M. (2024) · European Journal of Information Systems · doi:10.1080/0960085x.2024.2395531 · Cited by 26 · see citing works
Gender Bias in Artificial Intelligence: Empowering Women Through Digital Literacy
Shah, S. S. (2025) · Premier Journal of Artificial Intelligence · doi:10.70389/pjai.1000088 · Cited by 26 · see citing works
Gaba, A., Kaufman, Z., Cheung, J., Shvakel, M., Hall, K. W., Brun, Y., & Xiong, C. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327192 · Cited by 21 · see citing works
Dexe, J., Franke, U., Söderlund, K., van Berkel, N., Jensen, R. H., Lepinkäinen, N., & Vaiste, J. (2022) · The Geneva Papers on Risk and Insurance Issues and Practice · doi:10.1057/s41288-022-00271-9 · Cited by 19 · see citing works
Decuypere, A., & de Vijver, A. V. (2025) · Government Information Quarterly · doi:10.1016/j.giq.2024.102002 · Cited by 28 · see citing works
From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis
Li, Z., Zhu, H., Lu, Z., Xiao, Z., & Yin, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713133 · Cited by 18 · see citing works
Investigating and Mitigating Biases in Crowdsourced Data
Hettiachchi, D., Sanderson, M., Gonçalves, J., Hosio, S. J., Kazai, G., Lease, M., Schaekermann, M., & Yılmaz, E. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3481729 · Cited by 17 · see citing works
Self-citationJiang, L., Qin, X., Yam, K. C., Dong, X., Liao, W., & Chen, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0284840 · Cited by 19 · see citing works
Judging in the Dark: How Delivery Riders Form Fairness Perceptions Under Algorithmic Management
Xiang, Y., Du, J., zheng, X., Long, L., & Xie, H. Y. (2024) · Journal of Business Ethics · doi:10.1007/s10551-024-05883-w · Cited by 17 · see citing works
An Investigation of University Students' Attitudes Towards Artificial Intelligence Ethics
Aşıksoy, G. (2024) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v14i8.50769 · Cited by 25 · see citing works
Zhang, W.-Z., & Lian, R. (2025) · BMC Psychology · doi:10.1186/s40359-025-02741-w · Cited by 19 · see citing works
Ferguson, S. A., Aoyagui, P. A., Rizvi, R., Kim, Y., & Kuzminykh, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687056 · Cited by 15 · see citing works
Liu, Y., Lu, X., Li, C., & Zhao, G. (2022) · Sustainability · doi:10.3390/su14169900 · Cited by 14 · see citing works
Juijn, G., Stoimenova, N., Reis, J., & Nguyen, D. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604677 · Cited by 13 · see citing works
Weith, H., & Matt, C. (2023) · Electronic Markets · doi:10.1007/s12525-023-00668-x · Cited by 12 · see citing works
Aljuneidi, S., Heuten, W., Abdenebaoui, L., Wolters, M. K., & Boll, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642535 · Cited by 12 · see citing works
Xuan, Y., Small, E. A., Sokol, K., Hettiachchi, D., & Sanderson, M. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103376 · Cited by 12 · see citing works
Fairness perceptions of artificial intelligence decision-making
Jiang, L., Cao, L., Qin, X., Tan, L., Chen, C. H. E. N., & Xiaofei, P. E. N. G. (2022) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2022.01078 · Cited by 11 · see citing works
Method for Appropriating the Brief Implicit Association Test to Elicit Biases in Users
Dingler, T., Tag, B., Eccles, D., van Berkel, N., & Kostakos, V. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517570 · Cited by 11 · see citing works
Gemalmaz, M. A., & Yin, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534201 · Cited by 10 · see citing works
VanNostrand, P. M., Hofmann, D. M., Ma, L., & Rundensteiner, E. A. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658997 · Cited by 8 · see citing works
de Jong, S., Bos, M. W., van Berkel, N., & Lamers, M. H. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2535732 · Cited by 13 · see citing works
Students’ perspectives on using digital tools in programming courses
Asgari, M., Tsai, F.-C., Mannila, L., Strömbäck, F., & Sadique, K. M. (2024) · Discover Education · doi:10.1007/s44217-024-00144-4 · Cited by 10 · see citing works
Yurrita, M., Verma, H., Balayn, A., Gadiraju, U., Pont, S. C., & Bozzon, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713145 · Cited by 8 · see citing works
van Berkel, N., Tag, B., Jacobsen, R. M., Russo, D., Purchase, H. C., & Buschek, D. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103359 · Cited by 8 · see citing works
Self-citationDelaunay, J., Galárraga, L. A., Largouët, C., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711011 · Cited by 6 · see citing works
Welfarist Moral Grounding for Transparent AI
Narayanan, D. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593977 · Cited by 6 · see citing works
Langer, M., Baum, K. L., & Schlicker, N. (2023) · doi:10.31234/osf.io/ke256 · Cited by 6 · see citing works
Exploring the impact of explainable AI and cognitive capabilities on users’ decisions
Cau, F. M., & Spano, L. D. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09438-0 · Cited by 9 · see citing works
AI Ethics and Fairness in Machine Learning: Promoting Fairness in Critical Engineering Systems
Jonnalagadda, A. K., Naayini, P., Kamatala, S., Myakala, P. K., & Bura, C. (2025) · International Conference on Trends in Engineering Systems and Technologies (ICTEST) · doi:10.1109/ictest64710.2025.11042668 · Cited by 5 · see citing works
Do Crowdsourced Fairness Preferences Correlate with Risk Perceptions?
Haider, C. M. R., Clifton, C. W., & Yin, M. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645209 · Cited by 5 · see citing works
Algorithm Bias and Perceived Fairness: A Comprehensive Scoping Review
Saryazdi, A. H. (2024) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/3632634.3655848 · Cited by 6 · see citing works
Shrestha, H., Cachel, K., Alkhathlan, M., Rundensteiner, E. A., & Harrison, L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594108 · Cited by 4 · see citing works
AI Ethics in Engineering: Enhancing Fairness in Machine Learning Models for Critical Systems
Methuku, V., Aunugu, D. R., Jonnalagadda, A. K., & Myakala, P. K. (2025) · International Conference on Computational Innovations and Engineering Sustainability (ICCIES) · doi:10.1109/iccies63851.2025.11032595 · Cited by 4 · see citing works
Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability
Yurrita, M., Verma, H., Balayn, A., Alfrink, K., Gadiraju, U., & Bozzon, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757415 · Cited by 7 · see citing works
Precel, H., McDonald, A., Hecht, B. J., & Vincent, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642749 · Cited by 4 · see citing works
Humans or Machines for Teaching: Trust and Preferences among University Students
Asgari, M., Mannila, L., Tsai, F.-C., & Strömbäck, F. (2024) · International Conference on Education Technology and Computer · doi:10.1145/3702163.3702166 · Cited by 7 · see citing works
Yokota, T., & Nakao, Y. (2022) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3511047.3538033 · Cited by 4 · see citing works
Erlei, A., Cau, F. M., Georgiev, R., Kumar, S. C., Bizer, K., & Gadiraju, U. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791420 · Cited by 4 · see citing works
Guidelines for designing visualization tools for group fairness analysis in binary classification
Cruz, A., Salazar, T., Carvalho, M., Maç�ãs, C., Machado, P., & Abreu, P. H. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11179-w · Cited by 3 · see citing works
Making Smart Cities Explainable: What XAI Can Learn from the “Ghost Map”
Gupta, S., & Loukissas, Y. A. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585847 · Cited by 3 · see citing works
Anik, A. I., & Bunt, A. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00024 · Cited by 3 · see citing works
Xiao-ping, Z., & Cheng, X. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642280 · Cited by 5 · see citing works
Kaufman, Z., Endres, M., Xiong, C., & Brun, Y. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3634245 · Cited by 3 · see citing works
Leveraging Complementary AI Explanations to Mitigate Misunderstanding in XAI
Xuan, Y., Sokol, K., Sanderson, M., & Chan, J. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00034 · Cited by 2 · see citing works
How Does Predictive Information Affect Human Ethical Preferences?
Narayanan, S., Yu, G., Tang, W., Ho, C.-J., & Yin, M. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534165 · Cited by 2 · see citing works
"Good" XAI Design: For What? In Which Ways?
Wang, L., Liu, Y., & Goel, A. K. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720036 · Cited by 1 · see citing works
Anik, A. I., & Bunt, A. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789087 · Cited by 1 · see citing works
"Over-the-Hood" AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them
Anderson, A. W., Moussaoui, F., Guevara, J. N., Hamid, M. M., & Burnett, M. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789068 · Cited by 1 · see citing works
The process motivation model of algorithmic decision-making approach and avoidance
Caifeng, X., Wu, J., Liying, X. U., Feng, Y., ZHAND, Y., & Yingying, X. I. E. (2022) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2023.00060 · Cited by 1 · see citing works
de Santana, V. F., Quigley, L. T., Coates, J., & Graves, M. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3637130 · Cited by 1 · see citing works
A Scoping Review On The Impact Of Algorithm Bias On The Perceived FairnessPreprint
Saryazdi, A. H., & Mirhoseini, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3428999/v1 · Cited by 1 · see citing works
Fair Engineering of Machine Learning Systems – Lessons Learned from a Literature Review
Sengewald, J., & Lackes, R. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.605
Auditing Machine Behaviors: Does a diverse crowd result in different social norms?
Christoforou, E., Orphanou, K., Nicolaou, N., Stavrakis, E., & Otterbacher, J. (2023) · International Conference on Behavioral, Economic, and Socio-Cultural Computing · doi:10.1109/besc59560.2023.10386355
Yokota, T., & Nakao, Y. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-82633-7_8
User Reliance on AI Support for Collaborative Partner Selection
Hrkalović, T. M., Dudzik, B., Hao, C., & Willemsen, M. C. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789082
Exploring demographic influences on fairness perceptions across the ML development lifecycle
Mishra, A., Khazanchi, D., & Tripathi, A. (2026) · AI and Ethics · doi:10.1007/s43681-026-01034-1
Zipperling, D., Deck, L., Lanzl, J., & Kühl, N. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3858913
- Authors
- van Berkel, N., Goncalves, J., Russo, D., Hosio, S., & Skov, M. B.
- Affiliations
- Department of Computer Science Aalborg University, Denmark
- School of Computing and Information Systems The University of Melbourne, Australia
- Aalborg University (DK)
- University of Oulu (FI)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-13
- OpenAlex
- W3160735296
- Cited by
- 84
- References
- 61
- Citations by year
- 2026: 6 · 2025: 16 · 2024: 22 · 2023: 17 · 2022: 22 · 2021: 1
- FWCI
- 14.01
- Type
- conference paper
- Language
- en
- Open access
- green
- vbn.aau.dk
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- VBN Forskningsportal (Aalborg Universitet) (publishedVersion, other-oa)
- Minerva Access (University of Melbourne) (submittedVersion)
- VBN Forskningsportal (Aalborg Universitet) (submittedVersion)
- Funders
- Oulun Yliopisto (318930)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- algorithmic fairness, perceived fairness, machine learning predictors, visualization techniques, AI transparency, gender differences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
