Is this AI trained on Credible Data? The Effects of Labeling Quality and Performance Bias on User Trust
The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
To promote data transparency, frameworks such as CrowdWorkSheets encourage documentation of annotation practices on the interfaces of AI systems, but we do not know how they affect user experience. Will the quality of labeling affect perceived credibility of training data? Does the source of annotation matter? Will a credible dataset persuade users to trust a system even if it shows racial biases in its predictions? To find out, we conducted a user study (N = 430) with a prototype of a classification system, using a 2 (labeling quality: high vs. low) × 4 (source: others-as-source vs. self-as-source cue vs. self-as-source voluntary action, vs. self-as-source forced action) × 3 (AI performance: none vs. biased vs. unbiased) experiment. We found that high-quality labeling leads to higher perceived training data credibility, which in turn enhances users’ trust in AI, but not when the system shows bias. Practical implications for explainable and ethical AI interfaces are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · cited 1×
“According to a recent study, knowing that labelers are from diverse racial backgrounds can increase expectations of AI fairness and accuracy by triggering the representativeness heuristic [55], which further”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“Drawing on the definition of trust in automation [29], we define trust in AI as the extent to which users believe the AI system will take their welfare into account under conditions of uncertainty and vulnerability.”
BackgroundAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 1×
“Studies have shown that algorithm aversion can set in very quickly [14] and is quite difficult to overcome without enabling some user control over the algorithmic decision [15].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 1×
“This approach is echoed by the recently proposed documentation paradigm in XAI community, which advocates detailed reporting of training data and performance characteristics of machine learning models at the outset of model development [5, 19, 22, 41, 45].”
MethodDietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · cited 2×
“…an unintended consequence of machine learning, in which the model favors the dominant group in society over a minority group of individuals, and thereby violates the principle of group fairness, i.e., equal opportunity of positive classification outcomes regardless of race and other factors [56].”
BackgroundCredibility and trust of information in online environments: The use of cognitive heuristics
Metzger, M. J., & Flanagin, A. J. (2013) · Journal of Pragmatics · doi:10.1016/j.pragma.2013.07.012 · Cited by 1,000
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 1×
“This approach is echoed by the recently proposed documentation paradigm in XAI community, which advocates detailed reporting of training data and performance characteristics of machine learning models at the outset of model development [5, 19, 22, 41, 45].”
Method'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 2×
“Second, the positive mediation effect of labeling quality on AI trust shows the value of early communication in shaping user perceptions and trust in AI, which broadens current practices of XAI research that is dominated by post-hoc explanations (see examples [6, 16, 27, 31, 48, 53, 59]).”
BackgroundHow Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
Automation Bias: Decision Making and Performance in High-Tech Cockpits
Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0801_3 · Cited by 353
Ego involvement and attributions for success and failure.
Miller, D. T. (1976) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.34.5.901 · Cited by 337 · cited 1×
“As a result, when individuals feel involved in a task and responsible for the outcome, they tend to engage in self-serving bias [36], which may generate more favorable attitudes towards the labeling task, thus leading to higher perceived credibility of the training data.”
BackgroundExplaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286
A Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256
Wang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211
Smith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112 · cited 1×
“While it is important to have an accurate evaluation of racial bias in AI algorithms from the users’ perspective, an obvious follow-up question that AI developers and designers should be asking is: how to involve users to help create a better AI system, such that there is no bias in its performance in the future? Prio…”
BackgroundAnik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103 · cited 4×
“Aside from communicating training data demographics [2] and labelers’ racial backgrounds [9], designers can convey the extent to which the labels match the raw training data.”
BackgroundInfluentialRechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97 · cited 1×
“Given that accuracy is a straightforward way to evaluate AI performance and prior studies have shown the positive effect of AI accuracy on user perceptions and trust in AI [43], we predict that showing accurate labeling, i.”
BackgroundExploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86 · cited 2×
“Second, the positive mediation effect of labeling quality on AI trust shows the value of early communication in shaping user perceptions and trust in AI, which broadens current practices of XAI research that is dominated by post-hoc explanations (see examples [6, 16, 27, 31, 48, 53, 59]).”
BackgroundDíaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67 · cited 2×
“Second, the positive mediation effect of labeling quality on AI trust shows the value of early communication in shaping user perceptions and trust in AI, which broadens current practices of XAI research that is dominated by post-hoc explanations (see examples [6, 16, 27, 31, 48, 53, 59]).”
BackgroundIntroduction to Explainable AI
Liao, Q. V., Singh, M., Zhang, Y., & Bellamy, R. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3445016 · Cited by 34
Cited by
Chen, C., Lee, S., Jang, E., & Sundar, S. S. (2024) · Proceedings of the Second International Symposium on Trustworthy Autonomous Systems · doi:10.1145/3686038.3686060 · Cited by 21
Self-citationCao, H. J., Choi, K., Park, C., & Lee, H. R. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713173 · Cited by 22
“As AI replicates structural inequalities and historical biases in unexpected ways [20], the implications for how students understand AI and the content it provides could be signifcant [24, 33].”
Psychotherapists’ Trust, Distrust, and Generative AI Practices in Psychotherapy: Qualitative Study
Kuang, J., Pope, A. L., & Zhang, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/88932 · Cited by 7
Metadata
- Authors
- Chen, C., & Sundar, S. S.
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-11
- OpenAlex
- W4366549723
- Cited by
- 36
- References
- 28
- FWCI
- 9.28
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.