CoChat
Sign Up
Cited by 178Ethics and Social Impacts of AIHuman-Automation Interaction and SafetyPsychology of Moral and Emotional Judgment

How do people react to AI failure? Automation bias, algorithmic aversion, and perceived controllability

Open access
Jones-Jang, S Mo, Park, Yong Jin·2022·Journal of Computer-Mediated Communication
DOI 10.1093/jcmc/zmac029
Open this paper in CoChat — free accountOpen PDF

Abstract

AI can make mistakes and cause unfavorable consequences. It is important to know how people react to such AI-driven negative consequences and subsequently evaluate the fairness of AI’s decisions. This study theorizes and empirically tests two psychological mechanisms that explain the process: (a) heuristic expectations of AI’s consistent performance (automation bias) and subsequent frustration of unfulfilled expectations (algorithmic aversion) and (b) heuristic perceptions of AI’s controllability over negative results. Our findings from two experimental studies reveal that these two mechanisms work in an opposite direction. First, participants tend to display more sensitive responses to AI’s inconsistent performance and thus make more punitive assessments of AI’s decision fairness, when compared to responses to human experts. Second, as participants perceive AI has less control over unfavorable outcomes than human experts, they are more tolerant in their assessments of AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jcmc/zmac029

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 39 · Cited by 178

Research works printed but not linked: 35

  1. Alvarado-Valencia, J. A., & Barrero, L. H. (2014). Reliance, trust and heuristics in judgmental forecasting. Computers in Human Behavior, 36, 102-113. https://doi.org/10.1016/j.chb.2014.03.047 · DOI
  2. Araujo, T., Helberger, N., Kruikemeier, S., & De Vreese, C. H. (2020). In AI we trust? Perceptions about automated decision-making by ar- tificial intelligence. AI & Society, 35(3), 611-623. https://doi.org/10. 1007/s00146-019-00931-w · DOI · Paper page
  3. Banks, J. (2020). Optimus primed: Media cultivation of robot mental models and social judgments. Frontiers in Robotics and AI, 7, 62. https://doi.org/10.3389/frobt.2020.00062 · DOI
  4. Bucher, T. (2017). The algorithmic imaginary: Exploring the ordinary effects of Facebook algorithms. Information, Communication & Society, 20(1), 30-44. https://doi.org/10.1080/1369118X.2016. 1154086 · DOI
  5. Burton, J. W., Stein, M. K., & Jensen, T. B. (2020). A systematic review of algorithm aversion in augmented decision making. Journal of Behavioral Decision Making, 33(2), 220-239. https://doi.org/10. 1002/bdm.2155 · DOI · Paper page
  6. Campbell, M. C. (1999), Perceptions of price unfairness: Antecedents and consequences. Journal of Marketing Research, 36(2), 187-199. https://doi.org/10.1177/002224379903600 · DOI
  7. Cummings, M. L. (2017). Decision making in aviation. Routledge.
  8. Cloudy, J., Banks, J., & Bowman, N. D. (in press). The Str (AI) ght Scoop: Artificial intelligence cues reduce perceptions of hostile media bias. Digital Journalism, Online First. https://doi.org/10.1080/ 21670811.2021.1969974 · DOI
  9. Dawes Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist. 34(7), 571-582.
  10. Do ¨rr, K., & Hollnbuchner, K. (2017). Ethical challenges of algorithmic journalism. Digital Journalism, 5(4), 404-419. https://doi.org/10. 1080/21670811.2016.1167612 · DOI
  11. Garnham, A. (1987). Artificial intelligence: An introduction. Routledge.
  12. Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of infor- mal (subjective, impressionistic) and formal (mechanical, algorith- mic) pre-diction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2(2), 293-323. http://dx.doi. org/10.1037/1076-8971.2.2.293 · DOI · Paper page
  13. Guzman, A. L. (2020). Ontological boundaries between humans and computers and the implications for human-machine communication. Human-Machine Communication, 1, 37-54. https://doi.org/10. 30658/hmc.1.3 · DOI
  14. Helberger, N., Karppinen, K., & D'Acunto, L. (2018). Exposure diver- sity as a design principle for recommender systems. Information, Communication & Society, 21(2), 191-207. https://doi.org/10. 1080/1369118X.2016.1271900 · DOI
  15. Highhouse Scott. (2008). Stubborn Reliance on Intuition and Subjectivity in Employee Selection. Industrial and Organizational Psychology. 1(3), 333-342.
  16. Hsu, J. (2020). Can AI hiring systems be made antiracist? Makers and users of AI-assisted recruiting software reexamine the tools' develop- ment and how they're used-[News]. IEEE Spectrum, 57(9), 9-11. https://doi.org/10.1109/MSPEC.2020.9173891 · DOI
  17. Just, N., Latzer, M., & Warf, B. (2018). Algorithmic selection on the Internet. In B. Warf (Eds.), The SAGE encyclopedia of the internet. (pp.12-14). Sage.
  18. Klein, J., Moon, Y., & Picard, R. W. (2002). This computer responds to user frustration: Theory, design, and results. Interacting with Computers, 14(2), 119-140. https://doi.org/10.1016/S0953-5438(01)00053-4 · DOI
  19. Lee, E. J., & Nass, C. (2002). Experimental tests of normative group in- fluence and representation effects in computer-mediated communica- tion: When interacting via computers differs from interacting with computers. Human Communication Research, 28(3), 349-381. https://doi.org/10.1111/j.1468-2958.2002.tb00812.x · DOI
  20. Moon, W., Chung, M., & Jones-Jang, S. M. (in press). How can we fight partisan biases in the COVID-19 pandemic? AI source labels on fact-checking messages reduce motivated reasoning. Mass Communication & Society. Online First. https://doi.org/10.1080/ 15205436.2022.2097926 · DOI
  21. Noble, S. U. (2018). Algorithms of oppression. New York University Press.
  22. Paepcke, S., & Takayama, L. (2010). Judging a bot by its cover: An ex- periment on expectation setting for personal robots. Osaka, Japan: ACM/IEEE International Conference on Human-Robot Interaction (HRI). https://doi.org/10.1109/HRI.2010.5453268 · DOI
  23. Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people. Cambridge university press.
  24. Russell, S. J., & Norvig, P. (2009). Artificial intelligence: A modern ap- proach. Prentice Hall.
  25. Sandvig, C., Hamilton, K., Karaholios, K., & Langbort, C. (2016). Automation, algorithms, and politics j When the algorithm itself is a racist: diagnosing ethical harm in the basic components of software. International Journal of Communication, 10, 4972-4990. https:// ijoc.org/index.php/ijoc/article/view/6182
  26. Shank, D. B., & DeSanti, A. (2018). Attributions of morality and mind to artificial intelligence after real-world moral violations. Computers in human behavior, 86, 401-411. https://doi.org/10.1016/j.chb. 2018.05.014 · DOI
  27. Shin, D., & Park, Y. J. (2019). Role of fairness, accountability, and transparency in algorithmic affordance. Computers in Human Behavior, 98, 277-284. https://doi.org/10.1016/j.chb.2019.04.019 · DOI · Paper page
  28. Sundar, S. S., & Kim, J. (2019). Machine heuristic: When we trust com- puters more than humans with our personal information. Proceedings of the 2019 Conference on Human Factors in Computing Systems. Proceedings (CHI' 19), Paper No, 538. https:// doi.org/10.1145/3290605.3300768 · DOI · Paper page
  29. Stowers, K. (2017). The role of accounts and apologies in mitigating blame toward human and machine agents. Dissertation. https://stars. library.ucf.edu/etd/5932
  30. Thurman, N., Moeller, J., Helberger, N., & Trilling, D. (2019). My friends, editors, algorithms, and I. Digital Journalism, 7(4), 447-469. https://doi.org/10.1080/21670811.2018.1493936 · DOI
  31. Tversky, A., & Kahneman, D. (1985). The framing of decisions and the psychology of choice. In Behavioral decision making (pp. 25-41). Springer.
  32. van der Woerdt S., & Haselager P. (2016) Lack of effort or lack of abil- ity? Robot failures and human perception of agency and responsibil- ity. In T. Bosse & B Bredeweg (Eds.), BNAIC 2016: Artificial Intelligence. BNAIC 2016. Communications in Computer and Information Science (Vol. 765, pp. 155-168). Springer. https://doi. org/10.1007/978-3-319-67468-1_11 · DOI
  33. Westerman, D., Edwards, A. P., Edwards, C., Luo, Z., & Spence, P. R. (2020). I-it, i-thou, i-robot: The perceived humanness of AI in human-machine communication. Communication Studies, 71(3), 393-408. https://doi.org/10.1080/10510974.2020.1749683 · DOI
  34. Wojcieszak, M., Thakur, A., Ferreira Gonc¸alves, J. F., Casas, A., Menchen-Trevino, E., & Boon, M. (2021). Can AI enhance people's support for online moderation and their openness to dissimilar polit- ical views? Journal of Computer-Mediated Communication, 26(4), 223-243. https://doi.org/10.1093/jcmc/zmab006 · DOI
  35. Yang, K., & Stoyanovich, J. (2017). Measuring fairness in ranked out- puts. In Proceedings of the 29th International Conference on

Authors
Jones-Jang, S. M., & Park, Y. J.
Affiliations
  • Boston College (US)
  • Harvard University (US)
Corresponding author
Jones-Jang, S Mo
ORCID
Venue
Journal of Computer-Mediated Communication
Publisher
Oxford University Press (OUP)
Published
2022-11-04
Volume
28
Issue
1
ISSN
1083-6101 (ISSN-L)
DOI
10.1093/jcmc/zmac029
OpenAlex
W4309048115
Cited by
178
References
39
Citations by year
2026: 69 · 2025: 61 · 2024: 32 · 2023: 16
FWCI
31.59
Type
journal article
Language
en
Open access
Licence
Locations
Journal of Computer-Mediated Communication (publishedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
Keywords
AI failure, automation bias, algorithmic aversion, perceived controllability, human-AI comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review