Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems
The DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
AI technologies have been incorporated into many end-user applications. However, expectations of the capabilities of such systems vary among people. Furthermore, bloated expectations have been identified as negatively affecting perception and acceptance of such systems. Although the intelligibility of ML algorithms has been well studied, there has been little work on methods for setting appropriate expectations before the initial use of an AI-based system. In this work, we use a Scheduling Assistant - an AI system for automated meeting request detection in free-text email - to study the impact of several methods of expectation setting. We explore two versions of this system with the same 50% level of accuracy of the AI component but each designed with a different focus on the types of errors to avoid (avoiding False Positives vs. False Negatives). We show that such different focus can lead to vastly different subjective perceptions of accuracy and acceptance. Further, we design expectation adjustment techniques that prepare users for AI imperfections and result in a significant increase in acceptance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the success of many productive ML applications in real-world settings falls short of expectations [4].”
From Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2023) · cites this paper as [4]
In the citing paper
“Kocielnik et al. [29] found that AI systems that showcase subsets of errors, like false positives or false negatives, that have the same accuracy, can lead to vastly different perceptions of accuracy.”
From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [29]
In the citing paper
“People also expect AIs to be more competent and to act less immorally than humans [1,31,50,59], which could make the negative action more surprising.”
From The AI Double Standard: Humans Judge All AIs for the Actions of One (2025) · cites this paper as [31]
In the citing paper
“Accurately priming the user and adjusting their expectations to the system is difficult, and simply providing performance metrics is not meaningful to the average user, given their lack of familiarity with the inner workings of the technologies that they interact with [26, 30].”
From From "AI" to Probabilistic Automation: How Does Anthropomorphization of Technical Systems Descriptions Influence Trust? (2024) · cites this paper as [30]
Citations
Cites
(1975) · PhilPapers (PhilPapers Foundation) · Cited by 36,737
A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Understanding Information Systems Continuance: An Expectation-Confirmation Model1
Bhattacherjee, A. (2001) · MIS Quarterly · doi:10.2307/3250921 · Cited by 8,767 · cited 4×
“A number of theories related to expectations have been proposed [6, 37, 39].”
BackgroundMethodInfluentialAn Investigation into the Determinants of Customer Satisfaction
Churchill, G. A., & Surprenant, C. (1982) · Journal of Marketing Research · doi:10.2307/3151722 · Cited by 2,914
Oliver, R. L. (1977) · Journal of Applied Psychology · doi:10.1037/0021-9010.62.4.480 · Cited by 1,770
A Survey of Context-Aware Mobile Computing Research
Chen, G., & Kotz, D. F. (2000) · Cited by 1,749
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 1×
“In most cases, however, transparency techniques are intended for post fact explanations about the decisions of an AI or ML system [19, 22], rather than for adjusting user expectations of an AI-powered system prior to its use.”
BackgroundThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“However, most users do not expect their applications to behave inconsistently and imperfectly [15] which can lead to disap-pointment and potential abandonment of these technologies [41, 51] Previous work has shown that end-user expectations of technology are impacted by a variety of factors…”
BackgroundAn Experimental Study of Customer Effort, Expectation, and Satisfaction
Cardozo, R. N. (1965) · Journal of Marketing Research · doi:10.1177/002224376500200303 · Cited by 1,178
Levin, I. P., & Gaeth, G. J. (1988) · Journal of Consumer Research · doi:10.1086/209174 · Cited by 1,172 · cited 1×
“However, outside of work in marketing and advertising [11, 32, 38], few have explored how end-user expectations can be directly and explicitly shaped.”
BackgroundConsumer Dissatisfaction: The Effect of Disconfirmed Expectancy on Perceived Product Performance
Anderson, R. E. (1973) · Journal of Marketing Research · doi:10.1177/002224377301000106 · Cited by 1,066
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
A Theoretical Extension of the Technology Acceptance Model
VenkateshViswanath, & DavisFred, D. (2000) · Management Science · doi:10.5555/2786232.2786234 · Cited by 601
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 2×
“[35] decision tree path traversal was used to generate free-text description presented to the user).”
BackgroundMethodExplanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
Gregor, S., & Benbasat, I. (1999) · MIS Quarterly · doi:10.2307/249487 · Cited by 535 · cited 1×
“In most cases, however, transparency techniques are intended for post fact explanations about the decisions of an AI or ML system [19, 22], rather than for adjusting user expectations of an AI-powered system prior to its use.”
BackgroundVenkatesh, V., & Goyal, S. (2010) · MIS Quarterly · doi:10.2307/20721428 · Cited by 481
Overtrust of robots in emergency evacuation scenarios
Robinette, P., Li, W., Allen, R., Howard, A. M., & Wagner, A. R. (2016) · 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2016.7451740 · Cited by 468 · cited 1×
“…their applications to behave inconsistently and imperfectly [15] which can lead to disap-pointment and potential abandonment of these technologies [41, 51] Previous work has shown that end-user expectations of technology are impacted by a variety of factors including external information,…”
BackgroundUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · cited 12×
“(3) We show that an AI-powered system tuned to avoid False Positive errors can lead to lower average perception of accuracy and lower user acceptance than one tuned to avoid False Negatives (even when both versions perform at the same overall accuracy).”
BackgroundResultInfluentialConsumer Expectations, Product Performance, and Perceived Product Quality
Olshavsky, R. W., & Miller, J. A. (1972) · Journal of Marketing Research · doi:10.2307/3149600 · Cited by 347
Consumer Expectations, Product Performance, and Perceived Product Quality
Olshavsky, R. W., & Miller, J. A. (1972) · Journal of Marketing Research · doi:10.1177/002224377200900105 · Cited by 261
Assessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246 · cited 1×
“Lim et al. explored 7 intelligibility types aimed at communicating detailed mechanisms of how a simulated decision tree based machine learning (ML) algorithm in a hypothetical activity recognition system makes its decisions [33–35].”
BackgroundKay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · cited 1×
“While presence of visualization has been found beneficial, past work has also indicated that too complex visualizations (e.g., probability distributions) are too difficult for the everyday user to interpret [24, 26].”
BackgroundImproving Robot Controller Transparency Through Autonomous Policy Explanation
Hayes, B., & Shah, J. A. (2017) · Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020233 · Cited by 230
Reflection Companion: A Conversational System for Engaging Users in Reflection on Physical Activity
Kocielnik, R., Xiao, L., Avrahami, D., & Hsieh, G. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3214273 · Cited by 221 · cited 1×
“Specifically, the underlying algorithms driving AI functionalities, such as natural language understanding [12, 14, 30], sensorbased inferences [28, 29], web behavior prediction [27], or object recognition in video or images [25], are probabilistic and almost always operate at less than perfect accuracy.”
BackgroundSelf-citationInteracting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184
Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making
Fernandes, M., Walls, L., Munson, S., Hullman, J., & Kay, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173718 · Cited by 175
Graphical Communication of Uncertain Quantities to Nontechnical People
Ibrekk, H., & Morgan, M. G. (1987) · Risk Analysis · doi:10.1111/j.1539-6924.1987.tb00488.x · Cited by 159
Chen, N.-C., Drouhard, M., Kocielnik, R., Suh, J., & Aragon, C. R. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185515 · Cited by 152
Self-citationToolkit to support intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2010) · Proceedings of the 12th ACM international conference on Ubiquitous computing · doi:10.1145/1864349.1864353 · Cited by 143
Oliver Scheuer, A., & McLaren, B. M. (2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_618 · Cited by 110
Are explanations always important?: a study of deployed, low-cost intelligent interactive systems
Bunt, A., Lount, M., & Lauzon, C. (2012) · Proceedings of the 2012 ACM international conference on Intelligent User Interfaces · doi:10.1145/2166966.2166996 · Cited by 106 · cited 1×
“Most proposed approaches also require significant end-user interest and effort in understanding system behavior, which can be inefficient in many scenarios [7].”
BackgroundCalendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., & Monroy-Hernández, A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025780 · Cited by 106 · cited 1×
“Specifically, the underlying algorithms driving AI functionalities, such as natural language understanding [12, 14, 30], sensorbased inferences [28, 29], web behavior prediction [27], or object recognition in video or images [25], are probabilistic and almost always operate at less than perfect accuracy.”
BackgroundSelf-citationEnabling robots to communicate their objectives
Huang, S. H., Held, D., Abbeel, P., & Dragan, A. D. (2019) · Autonomous Robots · doi:10.1007/s10514-018-9771-0 · Cited by 101
Towards improving trust in context-aware systems by displaying system confidence
Antifakos, S., Kern, N., Schiele, B., & Schwaninger, A. (2005) · Proceedings of the 7th international conference on Human computer interaction with mobile devices & services · doi:10.1145/1085777.1085780 · Cited by 95
Too good to be bad: Favorable product expectations boost subjective usability ratings
Raita, E., & Oulasvirta, A. (2011) · Interacting with Computers · doi:10.1016/j.intcom.2011.04.002 · Cited by 90
Framing the user experience: information biases on website quality judgement
Hartmann, J., De Angeli, A., & Sutcliffe, A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357190 · Cited by 82
HIGHLIGHTS: Summarizing Agent Behavior to People
Amir, D., & Amir, O. (2018) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/wizc1785 · Cited by 76
Kujala, S., Mugge, R., & Miron-Shatz, T. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2016.09.011 · Cited by 74
Calendar.Help: Designing a Workflow-Based Scheduling Agent with Humans in the LoopPreprint
Cranshaw, J., Kocielnik, R., Yu, B., & Teevan, J. B. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2940295 · Cited by 66
Self-citationThe Illusion of Control: Placebo Effects of Control Settings
Vaccaro, K., Huang, D., Eslami, M., Sandvig, C., Hamilton, K., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173590 · Cited by 65
An Exploration of the Relation Between Expectations and User Experience
Michalco, J., Simonsen, J. G., & Hornbæk, K. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1065696 · Cited by 63
Survey-Based Assessment of Patients’ Understanding of Their Own Imaging Examinations
Rosenkrantz, A. B., & Flagg, E. R. (2015) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2015.02.006 · Cited by 51
Aeonium: Visual analytics to support collaborative qualitative coding
Drouhard, M., Chen, N.-C., Suh, J., Kocielnik, R., Peña-Araya, V., Cen, K., Zheng, X., & Aragon, C. (2017) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis.2017.8031598 · Cited by 38
Self-citationVisualization of uncertainty in context aware mobile applications
Rukzio, E., Hamard, J., Noda, C., & De Luca, A. (2006) · Proceedings of the 8th conference on Human-computer interaction with mobile devices and services · doi:10.1145/1152215.1152267 · Cited by 33
Explain Yourself: A Natural Language Interface for Scrutable Autonomous RobotsPreprint
Garcia, F. J. C., Robb, D. A., Liu, X., Laskov, A., Patrón, P., & Hastie, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02088 · Cited by 26
Verame, J. K. M., Costanza, E., & Ramchurn, S. D. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858369 · Cited by 15
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Simard, P., Chickering, D., Lakshmiratan, A., Charles, D., Bottou, L., Suarez, C. G. J., Grangier, D., Amershi, S., Verwey, J., & Suh, J. (2014) · arXiv · doi:10.48550/arxiv.1409.4814 · Cited by 15 · cited 1×
“5M messages from 150 users, we selected a subset of messages using a meeting request detection ML model trained using an interactive concept learning tool for training machine learning models similar to [44].”
MethodSelf-citationKocielnik, R., Keyes, O., Morgan, J. T., Taraborelli, D., McDonald, D. W., & Hsieh, G. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274360 · Cited by 11
Self-citationEnabling self-reflection with LifelogExplorer: generating simple views from complex data
Kocielnik, R., Maggi, F. M., & Sidorova, N. (2013) · TU/e Research Portal · doi:10.5555/2534504.2534544 · Cited by 10
Self-citationStress Analytics in Education.
Kocielnik, R., Pechenizkiy, M., & Sidorova, N. (2012) · TU/e Research Portal · Cited by 9
Self-citationAnthropometric Facial Emotion Recognition
Jarkiewicz, J., Kocielnik, R., & Marasek, K. P. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02577-8_21 · Cited by 8
Self-citation
Research works printed but not linked: 5
- T Bentley. 2000. Biasing Web Site User Evaluations: a study. In Proceedings of the Conference of the Computer Human Interaction Special Interest Group of the Ergonomics Society of Australia (OzCHI2000). 130-- 134.
- 10.5555/2906831.2906851 · doi:10.5555/2906831.2906851
- S Stumpf, A Bussone, and D O'Sullivan. 2016. Explanations considered harmful? user interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).
- 10.5555/2017458.2017463 · doi:10.5555/2017458.2017463
- Daniel S Weld and Gagan Bansal. 2018. Intelligible Artificial Intelligence. arXiv preprint arXiv:1803.04263 (2018). · arXiv
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766
“However, the success of many productive ML applications in real-world settings falls short of expectations [21].”
Background
Metadata
- Authors
- Kocielnik, R., Amershi, S., & Bennett, P. N.
- Affiliations
- University of Washington, Seattle, WA, USA
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-14
- OpenAlex
- W2918497321
- Cited by
- 277
- References
- 50
- Citations by year
- 2026: 49 · 2025: 48 · 2024: 46 · 2023: 37 · 2022: 29 · 2021: 42 · 2020: 22 · 2019: 3
- FWCI
- 19.43
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Decision Sciences › Information Systems and Management › Technology Adoption and User Behaviour
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- AI acceptance, expectation setting, false positives, false-negative results, perceived accuracy, user perceptions of AI, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.