Cited by 60Human-Automation Interaction and SafetyExplainable Artificial Intelligence (XAI)Ethics and Social Impacts of AI

Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making

Open access
Suresh, Harini, Lao, Natalie, Liccardi, Ilaria·2020·12th ACM Conference on Web Science

Abstract

ML decision-aid systems are increasingly common on the web, but their successful integration relies on people trusting them appropriately: they should use the system to fill in gaps in their ability, but recognize signals that the system might be incorrect. We measured how people’s trust in ML recommendations differs by expertise and with more system information through a task-based study of 175 adults. We used two tasks that are difficult for humans: comparing large crowd sizes and identifying similar-looking animals. Our results provide three key insights: (1) People trust incorrect ML recommendations for tasks that they perform correctly the majority of the time, even if they have high prior knowledge about ML or are given information indicating the system is not confident in its prediction; (2) Four different types of system information all increased people’s trust in recommendations; and (3) Math and logic skills may be as important as ML for decision-makers working with ML recommendations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3394231.3397922

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. Ifeoma Ajunwa. 2016. The Paradox of Automation as Anti-Bias Intervention. Forthcoming in Cardozo Law Review (2016).
  2. Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics. IEEE, 160-169.
  3. Erin Collins. 2018. Punishing Risk. Geo. LJ 107 (2018), 57.
  4. Dominik Dellermann, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, and Philipp Ebel. 2019. The Future of Human-AI Collaboration: A Taxon- omy of Design Knowledge for Hybrid Intelligence Systems. In Hawaii Interna- tional Conference on System Sciences (HICSS). Maui, Hawaii, USA. http://pubs.wi- kassel.de/wp-content/uploads/2018/09/JML_706.pdf
  5. Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144, 1 (2015), 114.
  6. Fei Jiang, Yong Jiang, Hui Zhi, Yi Dong, Hao Li, Sufeng Ma, Yilong Wang, Qiang Dong, Haipeng Shen, and Yongjun Wang. 2017. Artificial intelligence in health- care: past, present and future. Stroke and vascular neurology 2, 4 (2017), 230-243.
  7. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appro- priate reliance. Human factors 46, 1 (2004), 50-80.
  8. Weizhe Liu, Krzysztof Maciej Lis, Mathieu Salzmann, and Pascal Fua. 2019. Geo- metric and Physical Constraints for Drone-Based Head Plane Crowd Density Estimation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
  9. Weizhe Liu, Mathieu Salzmann, and Pascal Fua. 2019. Context-Aware Crowd Counting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  10. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wort- man Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
  11. Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender systems handbook. Springer, 353-382.
  12. Michael Yeomans, Anuj Shah, Sendhil Mullainathan, and Jon Kleinberg. 2017. Making sense of recommendations. Journal of Behavioral Decision Making (2017).

Cited by

Sorted by citations

Authors
Suresh, H., Lao, N., & Liccardi, I.
Affiliations
Massachusetts Institute of Technology (US)
Venue
12th ACM Conference on Web Science
Publisher
ACM
Published
2020-07-06
Pages
315-324
OpenAlex
W3027367391
Cited by
49
References
20
Citations by year
2026: 13 · 2025: 14 · 2024: 7 · 2023: 6 · 2022: 6 · 2021: 3
FWCI
5.29
Type
conference paper
Language
en
Open access
Locations
  • ACM Web Science Conference (WebSci) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
William and Flora Hewlett Foundation
Topics
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
trust in machine learning, human-AI decision-making, human expertise, system information, confidence measures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review