Don’t Just Tell Me, Ask Me: AI Systems that Intelligently Frame Explanations as Questions Improve Human Logical Discernment Accuracy over Causal AI explanations
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
Critical thinking is an essential human skill. Despite the importance of critical thinking, research reveals that our reasoning ability suffers from personal biases and cognitive resource limitations, leading to potentially dangerous outcomes. This paper presents the novel idea of AI-framed Questioning that turns information relevant to the AI classification into questions to actively engage users’ thinking and scaffold their reasoning process. We conducted a study with 204 participants comparing the effects of AI-framed Questioning on a critical thinking task; discernment of logical validity of socially divisive statements. Our results show that compared to no feedback and even causal AI explanations of an always correct system, AI-framed Questioning significantly increase human discernment of logically flawed statements. Our experiment exemplifies a future style of Human-AI co-reasoning system, where the AI becomes a critical thinking stimulator rather than an information teller.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Examples include asking individuals to make independent predictions before receiving AI input [16, 101, 105] or using 'slow algorithms' to reduce reliance on AI. Additionally, providing AI explanations without concrete recommendations [42] and using AI-framed questioning [25] have been shown to enhance critical thinking.”
From Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2025) · cites this paper as [25]
In the citing paper
“It enhances AI assistants by fostering deeper analytical thinking, encouraging users to form independent opinions before seeing AI suggestions (as in the Cognitive Forcing Function approach [16]) and stimulating critical thinking through AI-generated questions (similar to [25]).”
From Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2025) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“To generate the causal AI explanations in our study, we used the large language model "GPT-3"[6].”
MethodAn integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 3×
“This paper deploys the defnition of trust used in [40] as "the willingness of a [user] to be vulnerable to the actions of [an AI system] based on the expectation that the [AI system] will perform a particular action important to the [user], irrespective of the ability to monitor or control [the AI system]".”
BackgroundMethodCognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Perreault, K. (2011) · Manual Therapy · doi:10.1016/j.math.2010.09.003 · Cited by 4,759
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022 · cited 2×
“First, it has been shown to be more efective in changing people’s beliefs than if they are merely presented with corrective information[47].”
BackgroundBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 1×
“Since the reasoning of a system is often abstract and can be hard to understand, additional research into what makes a proper explanation is required so as to design and engineer AI generated explanations that are more natural and user friendly[9, 14, 33, 43].”
BackgroundPennycook, G., & Rand, D. G. (2020) · Journal of Personality · doi:10.1111/jopy.12476 · Cited by 879
Assessing miserly information processing: An expansion of the Cognitive Reflection Test
Toplak, M. E., West, R. F., & Stanovich, K. E. (2014) · Thinking & Reasoning · doi:10.1080/13546783.2013.844729 · Cited by 864 · cited 3×
“To measure the level of critical thinking of subjects, we used cognitive refection test (CRT), a task designed to measure a person’s ability to refect on a question and resist reporting the frst response that comes to mind [13].”
MethodInfluentialDesigning Experiments and Analyzing Data: A Model Comparison Perspective.
Serlin, R. C., Maxwell, S. E., & Delaney, H. D. (1991) · Journal of the American Statistical Association · doi:10.2307/2289744 · Cited by 812 · cited 1×
“Further post hoc tests with Benjamini-Hochberg correction were conducted to identify how intervention conditions difer from each other[39].”
Method
Research works printed but not linked: 20
- Roberto Confalonieri, Tarek R Besold, Tillman Weyde, Kathleen Creel, Tania Lombrozo, Shane Mueller, and Patrick Shafto. 2019. What makes a good explanation? Cognitive dimensions of explaining intelligent machines. CogSci 2019: Creativity+ Cognition+ Computation (2019).
- Edward M Glaser. 1985. Educating for responsible citizenship in a democracy. In National Forum, Vol. 65. Honor Society of Phi Kappa Phi, 24.
- Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. arXiv preprint arXiv:1707.06002(2017). · arXiv
- Jonathan Haidt. 2001. The emotional dog and its rational tail: a social intuitionist approach to moral judgment.Psychological review 108, 4 (2001), 814.
- Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
- Emily R Lai. 2011. Critical thinking: A literature review. Pearson’s Research Reports 6, 1 (2011), 40–41.
- David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The science of fake news. Science 359, 6380 (2018), 1094–1096.
- Jennifer S Lerner and Philip E Tetlock. 1999. Accounting for the effects of accountability.Psychological bulletin 125, 2 (1999), 255.
- Matthew D Lieberman. 2000. Intuition: a social cognitive neuroscience approach.Psychological bulletin 126, 1 (2000), 109.
- Mark McCaffrey, Paige Hayes, Jason Wagner, and Matt Hobbs. 2018. Consumer Intelligence Series: Prepare for the voice revolution. pwc. com (2018).
- Divyam Mehta, Aniket Dwivedi, Arunabha Patra, and M Anand Kumar. 2021. A transformer-based architecture for fake news classification. Social network analysis and mining 11, 1 (2021), 1–12.
- R Paul and L Elder. 2006. The art of socratic questioning. Dillon beach, CA: foundation for critical thinking. Psychology, Monograph Series II 3 (2006), 107–127.
- Gordon Pennycook, Ziv Epstein, Mohsen Mosleh, Antonio A Arechar, Dean Eckles, and David G Rand. 2021. Shifting attention to accuracy can reduce misinformation online. Nature 592, 7855 (2021), 590–595.
- Herbert A Simon. 1956. Rational choice and the structure of the environment.Psychological review 63, 2 (1956), 129.
- Herbert Alexander Simon. 1997. Models of bounded rationality: Empirically grounded economic reason. Vol. 3. MIT press.
- Jay J Van Bavel and Andrea Pereira. 2018. The partisan brain: An Identity-based model of political belief. Trends in cognitive sciences 22, 3 (2018), 213–224.
- Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. science 359, 6380 (2018), 1146–1151.
- Darrell M West. 2018. The future of work: Robots, AI, and automation. Brookings Institution Press.
- Cassandra Xia. 2014. A game-based intervention for the reduction of statistical cognitive biases. Ph. D. Dissertation. Massachusetts Institute of Technology.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems 32 (2019).
Cited by
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
A Roadmap to Pluralistic AlignmentPreprint
Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C. M., Ye, A., Jiang, L., Lu, X., Dziri, N., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05070 · Cited by 477
“…to represent multiple different perspectives can be valuable in creative production (Shanahan & Clarke, 2023), psychological inquiry (Shanahan et al., 2023), simulating social systems (Park et al., 2022), and deliberative discourse (Danry et al., 2023; Landemore & Page, 2015; Page, 2019; 2008).”
BackgroundAI makes you smarter but none the wiser: The disconnect between performance and metacognition
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108779 · Cited by 185
Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
“In this study, we explore whether LLMs can contribute to decision making processes by playing the role of devil’s advocate to encourage human decision makers to engage in constructive deliberation, through generating critique questions and comments to provoke human reasoning [21].”
BackgroundThe Role of Causality in Explainable Artificial Intelligence
Carloni, G., Berti, A., & Colantonio, S. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70015 · Cited by 127
Metadata
- Authors
- Danry, V., Pataranutaporn, P., Mao, Y., & Maes, P.
- Affiliations
- MIT Media Lab (US)
- Columbia University (US)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-13
- OpenAlex
- W4366590585
- Cited by
- 110
- References
- 38
- Citations by year
- 2026: 39 · 2025: 50 · 2024: 19 · 2023: 2
- FWCI
- 19.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- critical thinking, cognitive biases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Don’t Just Tell Me, Ask Me: AI Systems that Intelligently Frame Explanations as Questions Improve Human Logical Discernment Accuracy over Causal AI explanations. https://app.cochat.ai/papers/10.1145/3544548.3580672
- The paper
- Danry, V., Pataranutaporn, P., Mao, Y., & Maes, P. (2023). Don’t Just Tell Me, Ask Me: AI Systems that Intelligently Frame Explanations as Questions Improve Human Logical Discernment Accuracy over Causal AI explanations. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3580672
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.