Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 152 of the 152 references it cites were found in a registry or an index; 28 of the 145 sources its claims cite were checked for support.

Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions

Open access
Wischnewski, Magdalena, Krämer, Nicole, Müller, Emmanuel · 2023 · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Published version of a preprint

Abstract

Trust has been recognized as a central variable to explain the resistance to using automated systems (under-trust) and the overreliance on automated systems (over-trust). To achieve appropriate reliance, users’ trust should be calibrated to reflect a system’s capabilities. Studies from various disciplines have examined different interventions to attain such trust calibration. Based on a literature body of 1000+ papers, we identified 96 relevant publications which aimed to calibrate users’ trust in automated systems. To provide an in-depth overview of the state-of-the-art, we reviewed and summarized measurements of the trust calibration, interventions, and results of these efforts. For the numerous promising calibration interventions, we extract common design choices and structure these into four dimensions of trust calibration interventions to guide future studies. Our findings indicate that the measurement of the trust calibration often limits the interpretation of the effects of different interventions. We suggest future directions for this problem.

Overview

CoChat Summary

## TL;DR

This is a scoping review of empirical human-subject studies on trust calibration in automated systems, based on over 1000 papers screened down to 96 relevant publications. The paper finds that many interventions can influence trust, but measurement choices often make it hard to tell whether trust is actually calibrated to system capability.

## Key findings

  • The search screened a literature body of 1000+ papers and included 96 empirical studies; among these, 40 were online studies and 61 were lab studies.
  • Sample sizes ranged from 9 to 1224 participants in online studies (M = 232, SD = 224) and from 8 to 865 participants in lab studies (M = 72, SD = 125).
  • The authors identify four dimensions of trust calibration interventions: exo vs. endo, warranted vs. unwarranted, static vs. adaptive, and capabilities vs. process-oriented.
  • Results across interventions were mixed: uncertainty information, confidence displays, reliability updates, and some explanations sometimes helped calibration, but the paper also reports cases of increased workload, over-trust, or null effects; users were generally sensitive to reliability, yet decreases in reliability produced steeper trust decay than increases produced trust gain.

## Methods

  • Design: Scoping review following the PRISMA 2020 reporting standard; the authors state they did not intend a systematic review.
  • Data/sample: Peer-reviewed empirical studies with human subjects and a true trust calibration intervention, excluding preprints, theses, simulation-only studies, purely qualitative studies, and non-empirical theory/framework papers.
  • Analysis: Two-wave search across Google Scholar, major HCI conference proceedings, the ACM library, and Scopus; papers were coded for system, task, intervention, calibration measurement, trust measures, and calibration results, then grouped into categories such as task domains and calibration strategies.

## Limitations

  • The paper states that its results are limited to English-language texts.
  • The paper states that much of the corpus was screened by a single coder, though it reports an inter-rater agreement of Krippendorf’s alpha = .83 in the training phase.
  • The text makes evident that the review is limited by its operational definition of calibration as matching system capabilities to trust, which the authors acknowledge may leave out other trust factors such as process and purpose (stated in the paper).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3544548.3581197

Cites

152 of 152 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

14 Valid · 8 Not valid · 81 Not checked · 78 Own results and interpretations

70 more claims · 7 with their cited sources checked

Create a free account to see all 181 claims and to verify the 11 references still open.

See all claims — free account
  • It may state two separate claims as one. 65
  • Probably not a substantial claim. 53
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 111 references

Research works printed but not linked: 7

  1. Na Du, Kevin Y Huang, and X Jessie Yang. 2020. Not all information is equal: effects of disclosing different types of likelihood information on trust, compliance and reliance, and task performance in human-automation teaming. Human factors 62, 6 (2020), 987–1001.
  2. Alexandra D Kaplan, Theresa T Kessler, J Christopher Brill, and PA Hancock. 2021. Trust in artificial intelligence: Meta-analytic findings. Human Factors (2021), 00187208211013988.
  3. M. Madsen and S. Gregor. 2000. Measuring Human-Computer Trust. In Proceedings of 11th Australasian Conference on Information Systems. 6e8.
  4. Bonnie Marlene Muir. 2002. Operators’ trust in and use of automatic controllers in a supervisory process control task.(2002).
  5. Mona SharifHeravi, John R Taylor, Christopher J Stanton, Sandra Lambeth, and Christopher Shanahan. 2020. It’sa Disaster! Factors Affecting Trust Development and Repair Following Agent Task Failure. In Proceedings of the 2020 Australasian Conference on Robotics and Automation (ACRA 2020), 8-10 December 2020, Brisbane, Queensland.
  6. Yihan Wu and Ryan M Kelly. 2020. Online Dating Meets Artificial Intelligence: How the Perception of Algorithmically Generated Profile Text Impacts Attractiveness and Trust. In 32nd Australian Conference on Human-Computer Interaction. 444–453.
  7. Jin Xu and Ayanna Howard. 2022. Evaluating the Impact of Emotional Apology on Human-Robot Trust. In 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 1655–1661.

Cited by

See all 146

Metadata

Authors
Wischnewski, M., Krämer, N., & Müller, E.
Affiliations
  • Research Center for Trustworthy Data Science and Security, Germany
  • University of Duisburg-Essen (DE)
  • TU Dortmund University (DE)
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-16
OpenAlex
W4366548373
Cited by
146
References
111
Citations by year
2026: 73 · 2025: 43 · 2024: 23 · 2023: 7
FWCI
72.97
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Decision Sciences › Information Systems and Management › Technology Adoption and User Behaviour
Keywords
trust calibration, trust in automation, trust measurement, human-automation interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions. https://app.cochat.ai/papers/10.1145/3544548.3581197
The paper
Wischnewski, M., Krämer, N., & Müller, E. (2023). Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581197

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.