“I Think You Might Like This”: Exploring Effects of Confidence Signal Patterns on Trust in and Reliance on Conversational Recommender Systems
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
With the rapid growth of large language models, conversational recommender systems (CRSs) are on the rise. When receiving a CRS recommendation, one may encounter phrases of varying confidence levels such as “you might like...” or “I think you will like...,” but to the best of our knowledge, no work has investigated how the pattern of confidence signals from one recommendation to the next affects trust in and reliance on CRSs. In a mixed-methods user study, we explore how 30 participants interact with two Wizard of Oz music CRSs that grow increasingly confident, but only one uses natural language and color-coding to communicate confidence, which is accurate, random, always low, or always high. Through semi-structured interviews, survey responses, and recommendation ratings, we find evidence suggesting that an accurate confidence signal generates the greatest increase in trust-related metrics without encouraging over-reliance but potentially under-reliance. Furthermore, we identify design guidelines for CRS confidence signals associated with each trust-related metric: desire to use, perceived transparency, perceived ability, perceived benevolence, and perceived anthropomorphism.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
(2011) · Springer eBooks · doi:10.1007/978-0-387-85820-3 · Cited by 3,761
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 2×
“…[the song] like Above zero implies under-reliance because the final rating was higher than initially expected, while below zero implies over-reliance. it’s recommended to me That said, prior work has high-lighted how anthropomorphism can be interpreted as setting users up for disappointment [17].”
BackgroundGPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731
Proceedings of the Human Factors and Ergonomics Society 2019 Annual Meeting
(2019) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181319631171 · Cited by 1,432
Araujo, T. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.051 · Cited by 1,156 · cited 1×
“Though it is a significant area of interest in conversational AI [1, 2, 43], anthropomorphism has had a more complex relationship with trust in AI, as it can sometimes give rise to creepiness or disappointment when the system is not as capable as it seemed [17].”
BackgroundBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
Self-citationDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 3×
“Similarly, despite being a common affordance for aiding trust judgments [31], transparency has had mixed results with respect to improving trust in AI, sometimes leading to inappropriate trust [5, 24, 41].”
BackgroundResultDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Bias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 1×
“Similarly, despite being a common affordance for aiding trust judgments [31], transparency has had mixed results with respect to improving trust in AI, sometimes leading to inappropriate trust [5, 24, 41].”
ResultA Survey on Conversational Recommender Systems
Jannach, D., Manzoor, A., Cai, W., & Chen, L. (2022) · ACM Computing Surveys · doi:10.1145/3453154 · Cited by 521 · cited 2×
“We adopt Jannach et al.’s definition of a CRS: “ a software system that supports its users in achieving recommendation-related goals through a multi-turn dialogue ” [21].”
BackgroundMethodBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“While there are certainly other factors that could be measured in relation to trust such as control [23, 26] and predictability [4, 23], these perceived facets cover a large and informative portion of what may be influenced by CRS confidence signals.”
BackgroundThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 2×
“On the other hand, Bussone et al. saw only a slight impact on trust and reliance from the communication of confidence through percentages in a clinical decision support system [9].”
BackgroundTransparency and trust in artificial intelligence systems
Schmidt, P., Biessmann, F., & Teubner, T. (2020) · Journal of Decision Systems · doi:10.1080/12460125.2020.1819094 · Cited by 440
Explanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388
Qualification and certainty in L1 and L2 students' writing
Hyland, K., & Milton, J. (1997) · Journal of Second Language Writing · doi:10.1016/s1060-3743(97)90033-3 · Cited by 267
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · cited 1×
“Visual-izations, such as icon arrays and quantile dot plots, are also used to represent confidence [25, 37].”
MethodThe effect of social presence and chatbot errors on trust
Toader, D.-C., Boca, G., Toader, R., Măcelaru, M., Toader, C., Ighian, D., & Rădulescu, A. T. (2019) · Sustainability · doi:10.3390/su12010256 · Cited by 233 · cited 4×
“For system facets impacting trust, we measure perceived benevolence [49], ability [49], transparency [3, 39], and anthropomorphism [49], adapting questions from the works just cited.”
BackgroundMethodInfluentialTait, A. R., Voepel-Lewis, T., Zikmund-Fisher, B. J., & Fagerlin, A. (2010) · Journal of Health Communication · doi:10.1080/10810730.2010.492560 · Cited by 199
Automated diagnostic aids: The effects of aid reliability on users' trust and reliance
Wiegmann, D. A., Rich, A., & Zhang, H. (2001) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110110306 · Cited by 176 · cited 1×
“However, whereas confidence communicates the predicted accuracy of an individual output, reliability communicates the system’s overall predicted performance [12, 32, 52].”
BackgroundDesigning for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
Formats for Improving Risk Communication in Medical Tradeoff Decisions
Waters, E. A., Weinstein, N. D., Colditz, G. A., & Emmons, K. (2006) · Journal of Health Communication · doi:10.1080/10810730500526695 · Cited by 158
Bias and Debias in Recommender System: A Survey and Future DirectionsPreprint
Chen, J., Dong, H., Wang, X., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03240 · Cited by 154
2022 ACM Conference on Fairness Accountability and Transparency
(2022) · doi:10.1145/3531146 · Cited by 148
Crowd-Based Personalized Natural Language Explanations for Recommendations
Chang, S., Harper, F. M., & Terveen, L. (2016) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2959100.2959153 · Cited by 145 · cited 1×
“We measure reliance by taking final recommendation ratings as ground truth and compare those to initial ratings based on initial exposure to the recommendations, as in prior work [6, 10].”
MethodInspectability and control in social recommenders
Knijnenburg, B. P., Bostandjiev, S., O'Donovan, J., & Kobsa, A. (2012) · Proceedings of the sixth ACM conference on Recommender systems · doi:10.1145/2365952.2365966 · Cited by 137 · cited 1×
“While there are certainly other factors that could be measured in relation to trust such as control [23, 26] and predictability [4, 23], these perceived facets cover a large and informative portion of what may be influenced by CRS confidence signals.”
BackgroundHuman Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · cited 2×
“Agreement rate determines how often the human’s final decision is in agreement with the AI, while the switch rate determines how often the human agrees with the AI after having made a different initial decision [33, 40].”
BackgroundExplanations of recommendations
Tintarev, N. (2007) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/1297231.1297275 · Cited by 129 · cited 3×
“…did not improve participants trust in or acceptance of an art recommender [13], and Tintarev found that focus groups for a movie recommender did not like the idea of presenting confidence to justify a recommendation and preferred that uncertain recommendations not be presented to them at all [47].”
BackgroundResultMeasuring Recommendation Explanation Quality: The Conflicting Goals of Explanations
Balog, K., & Radlinski, F. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401032 · Cited by 99 · cited 3×
“For system facets impacting trust, we measure perceived benevolence [49], ability [49], transparency [3, 39], and anthropomorphism [49], adapting questions from the works just cited.”
BackgroundMethodResultExploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86 · cited 2×
“[50] Perceived Transparency I The song recommendations made sense to me.”
BackgroundMethodYou Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85
Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82
Confidence Displays and Training in Recommender Systems.
McNee, S. M., Lam, S. K., Guetzlaff, C., Konstan, J. A., & Riedl, J. (2003) · International Conference on Human-Computer Interaction · Cited by 74
Kangasrääsiö, A., Głowacka, D., & Samuel Kaski, A. (2015) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2678025.2701371 · Cited by 44 · cited 1×
“While there are certainly other factors that could be measured in relation to trust such as control [23, 26] and predictability [4, 23], these perceived facets cover a large and informative portion of what may be influenced by CRS confidence signals.”
BackgroundChen, J., Mishler, S., & Hu, B. (2021) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3051137 · Cited by 40
Conversational Agents Trust Calibration: A User-Centred Perspective to Design
Dubiel, M., Daronnat, S., & Leiva, L. A. (2022) · International Conference on Conversational User Interfaces · doi:10.1145/3543829.3544518 · Cited by 40
Uncertainty VisualizationPreprint
Padilla, L., Kay, M., & Hullman, J. R. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ebd6r · Cited by 31
Key Qualities of Conversational Recommender Systems: From Users’ Perspective
Jin, Y., Chen, L., Cai, W., & Pu, P. (2021) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3472307.3484164 · Cited by 28 · cited 1×
“Furthermore, in identifying key qualities of CRSs, Jin et al. list rapport (comparable to benevolence), perceived usefulness (comparable to ability), humanness, and transparency as important [22].”
BackgroundInvestigating confidence displays for top‐ N recommendations
Shani, G., Rokach, L., Shapira, B., Hadash, S., & Tangi, M. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22934 · Cited by 15
Proceedings of the 4th Conference on Conversational User Interfaces
(2022) · doi:10.1145/3543829 · Cited by 13
Lu, Y., & Sarter, N. (2019) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181319631345 · Cited by 5
Research works printed but not linked: 14
- Gavin Abercrombie, Amanda Cercas Curry, Mugdha Pandya, and Verena Rieser. 2021. Alexa, Google, Siri: What are your pronouns? Gender and anthropomorphism in the design and perception of conversational assistants. arXiv preprint arXiv:2106.02578 (2021). · arXiv
- Mustafa Bilgic and Raymond J Mooney. 2005. Explaining recommendations: Satisfaction vs. promotion. In Beyond personalization workshop, IUI, Vol. 5. 153.
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
- 10.1145/3449287 · doi:10.1145/3449287 · Paper page
- Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Van Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-adapted interaction 18, 5 (2008), 455–496.
- Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
- Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608 (2018). · arXiv
- Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021). · arXiv
- Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, and Sameer Singh. 2022. Rethinking Explainability as a Dialogue: A Practitioner’s Perspective. arXiv preprint arXiv:2202.01875 (2022). · arXiv
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- Alan M MacEachren, Robert E Roth, James O’Brien, Bonan Li, Derek Swingley, and Mark Gahegan. 2012. Visual semiotics & uncertainty visualization: An empirical study. IEEE transactions on visualization and computer graphics 18, 12 (2012), 2496–2505.
- Amy Rechkemmer and Ming Yin. 2022. When Confidence Meets Accuracy: Exploring the Effects of Multiple Performance Indicators on Trust in Machine Learning Models. In CHI Conference on Human Factors in Computing Systems. 1–14.
- Anuschka Schmitt, Thiemo Wambsganss, and Andreas Janson. 2022. Designing for Conversational System Trustworthiness: The Impact of Model Transparency on Trust and Task Performance. European Conference on Information Systems Research Papers (2022).
- Anna-Maria Seeger, Jella Pfeiffer, and Armin Heinzl. 2017. When do we need a human? Anthropomorphic design and trustworthiness of conversational agents. SIGHCI 2017 Proceedings (2017).
Cited by
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Lee, Y., Son, K., Kim, T. S., Kim, J., Chung, J. J. Y., Adar, E., & Kim, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3662681 · Cited by 36
Eckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 44
Explainability for Transparent Conversational Information-Seeking
Łajewska, W., Spina, D., Trippas, J. R., & Balog, K. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657768 · Cited by 26
“Some workers attempted to gauge system confidence by searching for implicit confidence signals like “I think” or “I believe” in the responses [39].”
Pareek, S., Schömbs, S., Velloso, E., & Gonçalves, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713468 · Cited by 27
Metadata
- Authors
- Radensky, M., Séguin, J. A., Lim, J. S., Olson, K., & Geiger, R.
- Affiliations
- University of Washington (US)
- Google (United States) (US)
- Creative Circle, USA
- Google (United Kingdom) (GB)
- Venue
- 2023 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2023-06-12
- Pages
- 792-804
- OpenAlex
- W4380367493
- Cited by
- 14
- References
- 41
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 5
- FWCI
- 3.60
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- conversational recommender systems, trust in AI, AI reliance, Wizard of Oz, music recommendation, large language models, color coding, perceived transparency, perceived benevolence, perceived anthropomorphism, perceived competence, design guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.