Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index; 29 of the 53 sources its claims cite were checked for support.

Investigating appropriate reliance on AI-Based decision support systems: the role of expertise, trust, and self-confidence

Open access
Dogru, Enis Ömer, Krämer, Nicole C. · 2025 · Journal of Decision Systems

Abstract

Achieving appropriate reliance on AI-based decision support systems (DSS) is a central challenge in human-AI decision-making. Both over-reliance and under-reliance can impair decision quality, underscoring the need to understand the psychological factors guiding reliance behaviour. This study examines how expertise, trust, and self-confidence relate to appropriate reliance within an AI-based DSS. Using structural equation modelling with data from 529 chess players with varying expertise-levels, we analysed direct, mediated, and moderated relationships among these factors. Results revealed no over-reliance, reflecting participants’ preference for self-reliance. Expertise was positively related to self-confidence and negatively related to trust, corresponding with a more critical stance toward AI advice. Trust was associated with reliance on accurate AI recommendations, whereas self-confidence was related to reliance on one’s own correct decisions. These findings support a relational framework linking user characteristics to behavioural reliance and inform the design of adaptive, explainable DSS that promote calibrated trust and effective decision-making.

Overview

CoChat Summary

TL;DR

This study models how expertise, trust, and self-confidence relate to appropriate reliance on an AI-based decision support system in chess. Using SEM with 529 chess players, it finds expertise is linked to lower trust and higher self-confidence, with trust mainly supporting reliance on correct AI advice and self-confidence supporting reliance on one’s own correct decisions.

Key findings

  • No over-reliance was observed: negative AI-reliance was rare (0.77% of cases), while negative self-reliance occurred frequently (13.22%), reflecting a preference for self-reliance.
  • Expertise positively predicted self-confidence (β = 0.663, p < .001) and negatively predicted trust (β = -0.524, p < .001); expertise also positively predicted RSR (β = 0.544, p < .001) but not RAIR (β = 0.039, p = .532).
  • Trust positively predicted RAIR (β = 0.233, p < .001) and negatively predicted RSR (β = -0.087, p = .023); self-confidence positively predicted RSR (β = 0.110, p = .010) but did not significantly predict RAIR (β = -0.066, p = .245).
  • Trust fully mediated the expertise–RAIR relationship (indirect β = -0.122, p < .001) and partially mediated expertise–RSR (indirect β = 0.046, p = .026); self-confidence partially mediated expertise–RSR (indirect β = 0.073, p = .011). The model explained 45.4% of RSR variance, 5.8% of RAIR variance, 31.2% of trust variance, and 44% of self-confidence variance.

Methods

  • Online mixed within-between-subjects experiment conducted via SoSciSurvey between 26 September and 15 October 2024; final analytic sample N = 529 chess players after exclusions, recruited via Prolific and chess clubs/forums/newsletters.
  • Expertise was measured continuously via 10 chess puzzles; reliance behaviour was assessed across 12 additional puzzles with GambitAI advice, where 4 of 12 recommendations were deliberately incorrect, yielding 67% system accuracy.
  • Structural equation modelling in R (lavaan) tested direct, mediated, and moderated relationships among expertise, trust, self-confidence, RAIR, and RSR, with z-standardised variables, bootstrapping (1,000 resamples), and five-fold cross-validation.

Limitations

  • The paper states the online design limited control over participants’ environments, and participants saw the same puzzles twice, potentially introducing learning effects or delayed insights.
  • The paper notes the one-minute puzzle time limit may have added unmeasured time pressure, and the AI’s low 67% accuracy likely encouraged cautious engagement and limits generalisability to higher-reliability settings.
  • The paper states age and years of chess experience were not included in the model, SEM relationships are correlational, and the low-stakes chess paradigm may limit generalisability to high-stakes or collaborative domains.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1080/12460125.2025.2593251

Cites

89 of 89 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY-NC-ND · From the PDF
Image 2
CC-BY-NC-ND · From the PDF
Image 3
CC-BY-NC-ND · From the PDF
Image 4
CC-BY-NC-ND · From the PDF
Image 5
CC-BY-NC-ND · From the PDF
Figure 1 Conceptual diagram of factors predicting appropriate reliance on AI-based DSS.
Figure 1 Conceptual diagram of factors predicting appropriate reliance on AI-based DSS.CC-BY-NC-ND · From the PDF
Image 7
CC-BY-NC-ND · From the PDF
Figure 3 Occurrence of positive and negative reliance behaviours for all expertise levels. Note . A horizontal and vertical jitter of 0.016 was added to visualise the accumulation of data at certain points.
Figure 3 Occurrence of positive and negative reliance behaviours for all expertise levels. Note . A horizontal and vertical jitter of 0.016 was added to visualise the accumulation of data at certain points.CC-BY-NC-ND · From the PDF
Figure 4 Final sem with standardised path coefficients. Note . *** p < .001; ** p < .01; * p < .05. Solid paths indicate significant relationships. Dashed paths indicate non-significant relationships.
Figure 4 Final sem with standardised path coefficients. Note . *** p < .001; ** p < .01; * p < .05. Solid paths indicate significant relationships. Dashed paths indicate non-significant relationships.CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

24 Valid · 2 Not valid · 22 Not checked · 115 Own results and interpretations

76 more claims · 3 with their cited sources checked

Create a free account to see all 163 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 59
  • Probably not a substantial claim. 32
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentr(529) = 0.884reported p < .001 · recomputed p < .001

    “r (529) = .884,p < .001”

  • Consistentr(529) = 0.737reported p < .001 · recomputed p < .001

    “r (529) = .737,p < .001”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 87 references

Cited by

See all 15

Metadata

Authors
Dogru, E. Ö., & Krämer, N. C.
Affiliations
University of Duisburg-Essen (DE)
Corresponding author
Dogru, Enis Ömer
Venue
Journal of Decision Systems
Published
2025-01-02
Volume
34
Issue
1
ISSN
  • 1246-0125 (ISSN-L)
  • 2116-7052
OpenAlex
W4416774727
Cited by
15
References
87
Citations by year
2026: 15
FWCI
16.62
Type
journal article
Language
en
Open access
Locations
  • Journal of Decision System (publishedVersion, cc-by-nc-nd)
  • Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
Funders
DFG Research Training Group
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Decision Sciences › Information Systems and Management › Technology Adoption and User Behaviour
Keywords
AI-based decision support systems, AI reliance, expertise, trust in AI, self-confidence, human-AI decision-making, chess players, explainable AI, adaptive decision support, trust calibration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Investigating appropriate reliance on AI-Based decision support systems: the role of expertise, trust, and self-confidence. https://app.cochat.ai/papers/10.1080/12460125.2025.2593251
The paper
Dogru, E. Ö., & Krämer, N. C. (2025). Investigating appropriate reliance on AI-Based decision support systems: the role of expertise, trust, and self-confidence. Journal of Decision Systems. https://doi.org/10.1080/12460125.2025.2593251

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.