Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 136 of the 136 references it cites were found in a registry or an index; 9 of the 117 sources its claims cite were checked for support.

Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

Ma, Shuai, Chen, Qiaoyi, Wang, Xinru, Zheng, Chengbo, Peng, Zhenhui, Yin, Ming, Ma, Xiaojuan · 2025 · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

Abstract

Traditional AI-assisted decision-making systems often provide fixed recommendations that users must either accept or reject entirely, limiting meaningful interaction - especially in cases of disagreement. To address this, we introduce Human-AI Deliberation, an approach inspired by human deliberation theories that enables dimension-level opinion elicitation, iterative decision updates, and structured discussions between humans and AI. At the core of this approach is Deliberative AI, an assistant powered by large language models (LLMs) that facilitates flexible, conversational interactions and precise information exchange with domain-specific models. Through a mixed-methods user study, we found that Deliberative AI outperforms traditional explainable AI (XAI) systems by fostering appropriate human reliance and improving task performance. By analyzing participant perceptions, user experience, and open-ended feedback, we highlight key findings, discuss potential concerns, and explore the broader applicability of this approach for future AI-assisted decision-making systems.

Overview

CoChat Summary

TL;DR

The paper proposes Human-AI Deliberation, an LLM-empowered Deliberative AI that integrates domain-specific models with LLMs to support dimension-level opinion elicitation, iterative updates, and structured human-AI discussion in AI-assisted decision-making. In an exploratory graduate-admissions study, Deliberative AI improved decision accuracy and reduced over-reliance compared with traditional XAI, though it lowered trust and satisfaction.

Key findings

  • Decision accuracy was significantly higher in Deliberative AI (M=0.598, SD=0.169) than in XAI (M=0.524, SD=0.16, p<0.05).
  • Deliberative AI showed lower agreement fraction (M=0.57, SD=0.24 vs. M=0.68, SD=0.27, p<0.05) and lower switch fraction (M=0.23, SD=0.35 vs. M=0.51, SD=0.41, p<0.001) than XAI; over-reliance was lower (M=0.47, SD=0.31 vs. M=0.65, SD=0.33, p<0.001), while under-reliance did not differ significantly (M=0.32, SD=0.28 vs. M=0.29, SD=0.30).
  • Deliberative AI participants reported lower trust than XAI participants (M=4.47, SD=1.68 vs. M=5.52, SD=1.27, p<0.01); no significant differences were found for helpfulness, understanding, mental demand, effort, or complexity, but satisfaction was significantly lower in Deliberative AI than in Human Alone. XAI participants reported higher confidence than Human Alone (M=6, SD=1.10 vs. M=5.59, SD=1.12, p<0.05) despite XAI final accuracy being lower than Human Alone.
  • Qualitatively, 43/48 Deliberative AI participants felt deliberation helped them make more informed decisions; 21/48 identified AI limitations, 15/48 reflected on and corrected their own mistakes, and 18/48 gained new knowledge. For AI updates, 25/48 said updates improved user experience, but 11/48 felt the AI updated too frequently and 13/48 were uncertain why or how the AI was updating.

Methods

  • Between-subjects mixed-methods study on Prolific: 174 recruited, 153 valid responses (Deliberative AI: 48, XAI: 51, Human Alone: 54). Participants had US graduate-admission experience and completed four main graduate-admission cases plus a tutorial.
  • The task used a synthesized dataset of 100 applicant profiles; a domain-specific linear regression classifier achieved 80% accuracy on a 30% test set. Study cases were selected from the test set and intentionally chosen to be ambiguous/conflict-prone, with human and AI individual performance between 50% and 60%.
  • Quantitative analysis used non-parametric tests (Mann-Whitney U; Kruskal-Wallis with Bonferroni correction); qualitative analysis used inductive thematic analysis of open-ended feedback and conversation logs. Pilot work included 30 participants and 226 deliberative statements; the Intention Analyzer reached 96% accuracy, and the Argument Evaluator showed Cohen’s κ=0.78 agreement with human scores.

Limitations

  • Stated: the college admissions task does not fully capture real-world admissions; SOP and recommendation-letter strength are reduced to scale values; participants were not actual admissions committee members; most had master’s rather than PhD admissions experience, so findings may not generalize to PhD admissions.
  • Stated: graduate admissions is subjective and lacks definitive ground truth; dataset labels came from a professional admissions committee, and accuracy plus over/under-reliance were used as objective metrics “to some extent.” The study used only four main tasks due to pilot fatigue, and the selected cases were intentionally ambiguous with low independent human/AI accuracy (50–60%), leaving less-ambiguous cases unaddressed.
  • Evident/mentioned: the AI opinion-update mechanism was not disclosed to participants, and some doubted its transparency or reliability; the LLM was GPT-3.5 and the domain model was a linear regression classifier, limiting claims about other models or tasks; the authors describe the study as exploratory rather than an effectiveness test for graduate admissions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3706598.3713423

Cites

136 of 136 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 3 Not valid · 51 Not checked · 137 Own results and interpretations

73 more claims

Create a free account to see all 194 claims and to verify the 103 references still open.

See all claims — free account
  • It may state two separate claims as one. 68
  • Probably not a substantial claim. 75

Create a free account to see these claims.

Citations

Cites

Show all 119 references

Research works printed but not linked: 8

  1. André Bächtiger, Susumu Shikano, Seraina Pedrini, and Mirjam Ryser. 2009. Measuring deliberation 2.0: standards, discourse types, and sequenzialization. In ECPR General Conference. Potsdam, 5–12.
  2. Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
  3. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).
  4. Fred C Lunenburg. 2010. The decision making process.. In National Forum of Educational Administration & Supervision Journal , Vol. 27.
  5. Shuai Ma Mingfei Sun and Xiaojuan Ma. 2022. Modeling Adaptive Expression of Robot Learning Engagement and Exploring its Effects on Human Teachers. ACM Transactions on Computer-Human Interaction (2022).
  6. Philip E Tetlock. 2017. Expert political judgment. In Expert Political Judgment. Princeton University Press.
  7. Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Fei Xia Ed Chi Quoc V Le Denny Zhou et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022) 24824–24837.
  8. Jieqiong Zhao Yixuan Wang Michelle V Mancenido Erin K Chiou and Ross Maciejewski. 2023. Evaluating the impact of uncertainty visualization on model reliance. IEEE Transactions on Visualization and Computer Graphics (2023).

Cited by

See all 49

Metadata

Authors
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X.
Affiliations
  • Hong Kong University of Science and Technology (HK)
  • Purdue University West Lafayette (US)
  • Sun Yat-sen University (CN)
Venue
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2025-04-25
Pages
1-23
OpenAlex
W4409736269
Cited by
48
References
119
Citations by year
2026: 42 · 2025: 6
FWCI
45.52
Type
conference paper
Language
en
Open access
closed
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
Funders
Research Grants Council of the Hong Kong Special Administrative Region under General Research Fund (GRF) (16207923)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
AI-assisted decision-making, large language models, human-AI interaction, explainable AI, conversational AI assistant, user experience

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making. https://app.cochat.ai/papers/10.1145/3706598.3713423
The paper
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025). Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3713423

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.