AI Trust Score: A User-Centered Approach to Building, Designing, and Measuring the Success of Intelligent Workplace Features
Abstract
This paper describes how to help product teams move increasingly towards responsible AI experiences by examining user trust within business contexts. We discuss the development, application, and validation of the AI Trust Score, which allows teams to measure the success of an AI feature from enterprise users’ perspectives. The Trust Score is a multi-dimensional metric which consists of several statements users respond to that inform their trust, including whether an AI feature helps with job efficiency and effectiveness and understanding how and when to use the feature for their job role. The metric, along with usability feedback, can be adapted to encourage UX-related conversations and actions within product or feature development processes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability; support for traceability and auditing; and strategies to address concerns within existing organizational structures and dynamics.
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [71]
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1× · see citing works
“Because trust is a complex social process [13–15], we are also examining additional factors that may infuence trust beyond a user’s interaction with an AI feature, such as brand perception (i.e., opinion of company and tools) and AI/ML profciencies (e.g., understanding of probability and…”
BackgroundZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 1× · see citing works
“Furthermore, there are an increasing number of frameworks and guidance on how to approach UX research on AI experiences [8, 9].”
Background
Flagged first, then by citations
Cited by
Sorted by citations
The explainability paradox: Challenges for xAI in digital pathology
Evans, T., Retzlaff, C. O., Geißler, C., Kargl, M., Plass, M., Müller, H., Kiehl, T.-R., Zerbe, N., & Holzinger, A. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.03.009 · Cited by 135 · see citing works
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101 · see citing works
Will AI Console Me when I Lose my Pet? Understanding Perceptions of AI-Mediated Email Writing
Liu, Y., Mittal, A., Yang, D., & Bruckman, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517731 · Cited by 92 · see citing works
Believe in artificial intelligence? A user study on the chatGPT’s fake information impact.
Amaro, I., Barra, P., Greca, A. D., Francese, R., & Tucci, C. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3291539 · Cited by 60 · see citing works
- Authors
- Wang, J., & Moulden, A.
- Affiliations
- Microsoft (United States) (US)
- ORCID
- Wang, Jennifer
- Venue
- Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-08
- Pages
- 1-7
- OpenAlex
- W3162540384
- Cited by
- 18
- References
- 3
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 2 · 2023: 5 · 2022: 4
- FWCI
- 2.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Decision Sciences › Information Systems and Management › Technology Adoption and User Behaviour
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- user trust in AI, responsible AI, enterprise AI adoption, trust metrics, work efficiency, work effectiveness, usability feedback, user-centered design, product development process
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
