Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
Open accessAbstract
Large language models (LLMs) are trained to imitate humans to explain human decisions. However, do LLMs explain themselves? Can they help humans build mental models of how LLMs process different inputs? To answer these questions, we propose to evaluate $\textbf{counterfactual simulatability}$ of natural language explanations: whether an explanation can enable humans to precisely infer the model's outputs on diverse counterfactuals of the explained input. For example, if a model answers "yes" to the input question "Can eagles fly?" with the explanation "all birds can fly", then humans would infer from the explanation that it would also answer "yes" to the counterfactual input "Can penguins fly?". If the explanation is precise, then the model's answer should match humans' expectations. We implemented two metrics based on counterfactual simulatability: precision and generality. We generated diverse counterfactuals automatically using LLMs. We then used these metrics to evaluate state-of-the-art LLMs (e.g., GPT-4) on two tasks: multi-hop factual reasoning and reward modeling. We found that LLM's explanations have low precision and that precision does not correlate with plausibility. Therefore, naively optimizing human approvals (e.g., RLHF) may not be a sufficient solution.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The counterfactual simulatability framework aims to measure whether model explanations on one input help humans predict what predictions models will give on other inputs.
“The counterfactual simulatability framework of explanation faithfulness aims to measure whether model explanations on one input help humans predict what predictions models will give on other inputs (Doshi-Velez and Kim, 2017;Hase et al., 2020;Chen et al., 2023).”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Chen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1× · see citing works
“(Papineni et al., 2002) 2.”
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 2× · see citing works
“, 2023) or reinforcement learning (Schulman et al., 2017) by directly optimizing the simulatability score calculated by LLM simulators.”
BackgroundMethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1× · see citing works
“In Chainof-Thought (CoT), given an input x, the model first generates a reasoning ex, and then predicts the output ox conditioned on x and ex (Nye et al.; Wei et al., 2022; Wang et al., 2023).”
MethodTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 3× · see citing works
“Simulatability measures how well humans can predict the model’s outputs based on its explanations (Doshi-Velez and Kim, 2017; Ribeiro et al., 2018; Chandrasekaran et al., 2018; Hase and Bansal, 2020); in particular, simulatability is a special case of faithfulness, which requires the output…”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · see citing works
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1× · see citing works
“It is different from faithfulness, which measures whether an explanation is consistent with the model’s own de- cision process (Harrington et al., 1985; Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Lakkaraju et al., 2019; Jacovi and Goldberg, 2020).”
ResultRoss, R. (2004) · SIGA · doi:10.1145/992287.992306 · Cited by 2,444 · cited 1×
“Humans can use a model’s explanations to build mental models of how the model behaves on various inputs (Johnson-Laird, 1980; Collins and Gentner, 1987; Garnham, 1987; Gentner and Stevens, 2014; Bansal et al., 2019).”
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · see citing works
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 4× · see citing works
“(ii) Counterfactual simulatability does not correlate with plausibility, and thus RLHF methods that make humans happy may not improve counterfactual simulatability.”
BackgroundInfluentialMental Models in Cognitive Science
Johnson-Laird, P. (1980) · Cognitive Sciences · doi:10.1207/s15516709cog0401_4 · Cited by 984 · cited 2× · see citing works
“Humans can use a model’s explanations to build mental models of how the model behaves on various inputs (Johnson-Laird, 1980; Collins and Gentner, 1987; Garnham, 1987; Gentner and Stevens, 2014; Bansal et al., 2019).”
MethodSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503 · cited 2× · see citing works
“It is different from faithfulness, which measures whether an explanation is consistent with the model’s own de- cision process (Harrington et al., 1985; Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Lakkaraju et al., 2019; Jacovi and Goldberg, 2020).”
BackgroundResultBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 3× · see citing works
“…an AI system is important, as it helps humans understand what an AI system can and cannot achieve (Chandrasekaran et al., 2018), which informs humans how to improve the system or appropriately deploy the system without misuse or overtrust (Cassidy, 2009; Bansal et al., 2019; Ye and Durrett, 2022).”
BackgroundTraining a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 3× · see citing works
“Building mental models also detects if the model biases against specific groups of people (Vig et al., 2020; Ravfogel et al., 2020) or encourages illegal behaviors against human values (Hendrycks et al., 2021; Bai et al., 2022).”
BackgroundMultimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418 · see citing works
Understanding Dataset Difficulty with V-Usable InformationPreprint
Ethayarajh, K., Choi, Y., & Swayamdipta, S. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2110.08420 · Cited by 376 · cited 3× · see citing works
“We evaluate explanations on multi-hop reasoning (StrategyQA) and reward modeling (Stanford Human Preference).”
Methode-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367 · cited 1× · see citing works
“In Post-Hoc, given an input x, the model first predicts the output ox, and then generates an explanation ex conditioned on x and ox (Camburu et al., 2018; Park et al., 2018).”
MethodCounterfactual Visual ExplanationsPreprint
Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07451 · Cited by 326 · cited 1× · see citing works
“As modern AI models get stronger performance on more tasks, humans can learn difficult tasks by forming mental models of AI models (Mac Aodha et al., 2018; Goyal et al., 2019).”
BackgroundNull It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287 · cited 2× · see citing works
“Building mental models also detects if the model biases against specific groups of people (Vig et al., 2020; Ravfogel et al., 2020) or encourages illegal behaviors against human values (Hendrycks et al., 2021; Bai et al., 2022).”
BackgroundTurpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · see citing works
Faithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314229 · Cited by 261 · cited 1× · see citing works
“It is different from faithfulness, which measures whether an explanation is consistent with the model’s own de- cision process (Harrington et al., 1985; Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Lakkaraju et al., 2019; Jacovi and Goldberg, 2020).”
ResultEvaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Hase, P., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.491 · Cited by 254 · cited 2× · see citing works
“…well humans can predict the model’s outputs based on its explanations (Doshi-Velez and Kim, 2017; Ribeiro et al., 2018; Chandrasekaran et al., 2018; Hase and Bansal, 2020); in particular, simulatability is a special case of faithfulness, which requires the output predictor to be humans rather…”
BackgroundBeware explanations from AI in health care
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021) · Science · doi:10.1126/science.abg1834 · Cited by 252 · cited 2× · see citing works
“…know when and how to use the model without misuse and overtrust, especially in high-stakes domains such as healthcare (Adadi and Berrada, 2020; Merry et al., 2021; Babic et al., 2021), legal (Deeks, 2019; Norkute et al., 2021), and law enforcement (Matulionyte and Hanif, 2021; Hall et al., 2022).”
Background<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · cited 2× · see citing works
“StrategyQA is a multi-hop question-answering dataset on open-domain questions (Geva et al., 2021).”
BackgroundMethodLarge Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · see citing works
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. S. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.523 · Cited by 210 · see citing works
Contrastive explanation: a structural-model approach
Miller, T. (2021) · The Knowledge Engineering Review · doi:10.1017/s0269888921000102 · Cited by 186 · cited 1× · see citing works
“One possible solution is contrastive simulation (Jacovi et al., 2021; Miller, 2021; Yin and Neubig, 2022), where a human simulator is shown the model’s output mixed with fake outputs (distractors) and selects which output is from the model based on the explanation.”
BackgroundMental models as representations of discourse and text
Franz, A., & Garnham, A. (1989) · doi:10.2307/1423060 · Cited by 166
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12813 · Cited by 156 · cited 1× · see citing works
“…evaluation pipeline (Table 3), one possible way to improve counterfactual simulatability is via self-training (Huang et al., 2022; Weng et al., 2022; Peng et al., 2023) or reinforcement learning (Schulman et al., 2017) by directly optimizing the simulatability score calculated by LLM simulators.”
BackgroundSelf-citationNarayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · cited 1× · see citing works
“Plausibility evaluates humans’ preference of an explanation based on its factual correctness and logical coherence (Herman, 2017; Lage et al., 2019; Jacovi and Goldberg, 2020).”
BackgroundShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · see citing works
Towards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 143 · see citing works
Harvey Friedman's Research on the Foundations of Mathematics
Friedman, H., & Harrington, L. (1985) · Studies in logic and the foundations of mathematics · doi:10.1016/s0049-237x(09)x7008-4 · Cited by 143 · cited 1× · see citing works
“It is different from faithfulness, which measures whether an explanation is consistent with the model’s own de- cision process (Harrington et al., 1985; Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Lakkaraju et al., 2019; Jacovi and Goldberg, 2020).”
ResultWT5?! Training Text-to-Text Models to Explain their PredictionsPreprint
Narang, S., Raffel, C., Lee, K., Roberts, A., Fiedel, N., & Malkan, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14546 · Cited by 141 · cited 2× · see citing works
“explanation, the simulation input is the explained input (Hase et al., 2020; Narang et al., 2020; Wiegreffe et al., 2021; Chen et al., 2022; Chan et al., 2022).”
BackgroundExplainable AI for Healthcare: From Black Box to Interpretable Models
Adadi, A., & Berrada, M. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-981-15-0947-6_31 · Cited by 137 · cited 1× · see citing works
“model’s capacity and limitations so that users know when and how to use the model without misuse and overtrust, especially in high-stakes domains such as healthcare (Adadi and Berrada, 2020; Merry et al., 2021; Babic et al., 2021), legal (Deeks, 2019;”
BackgroundMeasuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.804 · Cited by 119 · cited 1× · see citing works
“explanation, the simulation input is the explained input (Hase et al., 2020; Narang et al., 2020; Wiegreffe et al., 2021; Chen et al., 2022; Chan et al., 2022).”
BackgroundNILE : Natural Language Inference with Faithful Natural Language Explanations
Kumar, S., & Talukdar, P. P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.771 · Cited by 114 · cited 1× · see citing works
“In prior work, faithfulness is usually evaluated by whether it is possible to train a black-box model to predict the model’s outputs based on its explanations (Li et al., 2020; Kumar and Talukdar, 2020; Lyu et al., 2022).”
BackgroundLarge Language Models are Better Reasoners with Self-Verification
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., & Zhao, J. (2022) · arXiv · doi:10.48550/arxiv.2212.09561 · Cited by 110 · cited 1× · see citing works
“…simulators in the evaluation pipeline (Table 3), one possible way to improve counterfactual simulatability is via self-training (Huang et al., 2022; Weng et al., 2022; Peng et al., 2023) or reinforcement learning (Schulman et al., 2017) by directly optimizing the simulatability score calculated by…”
BackgroundContrastive Explanations for Model Interpretability
Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., & Goldberg, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.120 · Cited by 83 · cited 2× · see citing works
“One possible solution is contrastive simulation (Jacovi et al., 2021; Miller, 2021; Yin and Neubig, 2022), where a human simulator is shown the model’s output mixed with fake outputs (distractors) and selects which output is from the model based on the explanation.”
BackgroundAlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2305.14387 · Cited by 78 · see citing works
Faithful Multimodal Explanation for Visual Question Answering
Wu, J., & Mooney, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4812 · Cited by 73 · cited 1× · see citing works
“It is different from faithfulness, which measures whether an explanation is consistent with the model’s own de- cision process (Harrington et al., 1985; Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Lakkaraju et al., 2019; Jacovi and Goldberg, 2020).”
ResultThe Promise and Peril of Human Evaluation for Model InterpretabilityPreprint
Herman, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07414 · Cited by 69 · see citing works
Explainable artificial intelligence for digital forensics
Hall, S. W., Sakzad, A., & Choo, K. R. (2022) · WIREs Forensic Science · doi:10.1002/wfs2.1434 · Cited by 66 · cited 1× · see citing works
“…know when and how to use the model without misuse and overtrust, especially in high-stakes domains such as healthcare (Adadi and Berrada, 2020; Merry et al., 2021; Babic et al., 2021), legal (Deeks, 2019; Norkute et al., 2021), and law enforcement (Matulionyte and Hanif, 2021; Hall et al., 2022).”
BackgroundHase, P., Zhang, S., Xie, H., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.390 · Cited by 62 · cited 2× · see citing works
“In prior work that evaluates the simulatability of a natural language explanation, the simulation input is the explained input (Hase et al., 2020; Narang et al., 2020; Wiegreffe et al., 2021; Chen et al., 2022; Chan et al., 2022).”
BackgroundDo explanations make VQA models more predictable to a human?
Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., & Parikh, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1128 · Cited by 59 · cited 4× · see citing works
“Simulatability measures how well humans can predict the model’s outputs based on its explanations (Doshi-Velez and Kim, 2017; Ribeiro et al., 2018; Chandrasekaran et al., 2018; Hase and Bansal, 2020); in particular, simulatability is a special case of faithfulness, which requires the output…”
BackgroundInfluentialTeaching Categories to Human Learners with Visual Explanations
Aodha, O. M., Su, S., Chen, Y., Perona, P., & Yue, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00402 · Cited by 52 · see citing works
Self-citationInterpreting Language Models with Contrastive Explanations
Yin, K., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.14 · Cited by 45 · see citing works
Joshi, B., Liu, Z.-Y., Ramnath, S., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.07095 · Cited by 44
Faithful Reasoning Using Large Language Models
Creswell, A., & Shanahan, M. (2022) · arXiv · doi:10.48550/arxiv.2208.14271 · Cited by 37 · cited 1× · see citing works
“soning (CoT/Post-Hoc explanations) that doesn’t correspond to their internal reasoning (Turpin et al., 2023; Creswell and Shanahan, 2022), but further experiments are needed to study this observation.”
BackgroundNorkutė, M., Herger, N., Michalak, L., Mulder, A., & Gao, S. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3443441 · Cited by 33 · cited 1× · see citing works
“…know when and how to use the model without misuse and overtrust, especially in high-stakes domains such as healthcare (Adadi and Berrada, 2020; Merry et al., 2021; Babic et al., 2021), legal (Deeks, 2019; Norkute et al., 2021), and law enforcement (Matulionyte and Hanif, 2021; Hall et al., 2022).”
BackgroundREV: Information-Theoretic Evaluation of Free-Text RationalesPreprint
Chen, H., Brahman, F., Ren, X., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2210.04982 · Cited by 31 · cited 2× · see citing works
“explanation, the simulation input is the explained input (Hase et al., 2020; Narang et al., 2020; Wiegreffe et al., 2021; Chen et al., 2022; Chan et al., 2022).”
BackgroundA mental models approach for defining explainable artificial intelligence
Merry, M., Riddle, P., & Warren, J. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01703-7 · Cited by 29 · cited 1× · see citing works
“…users know when and how to use the model without misuse and overtrust, especially in high-stakes domains such as healthcare (Adadi and Berrada, 2020; Merry et al., 2021; Babic et al., 2021), legal (Deeks, 2019; Norkute et al., 2021), and law enforcement (Matulionyte and Hanif, 2021; Hall et al.,…”
BackgroundEvaluating Explanation Methods for Neural Machine Translation
Li, J., Liu, L., Li, H., Li, G., Huang, G., & Shi, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.35 · Cited by 25 · cited 1× · see citing works
“In prior work, faithfulness is usually evaluated by whether it is possible to train a black-box model to predict the model’s outputs based on its explanations (Li et al., 2020; Kumar and Talukdar, 2020; Lyu et al., 2022).”
BackgroundHILDIF: Interactive Debugging of NLI Models Using Influence Functions
Zylberajch, H., Lertvittayakumjorn, P., & Toni, F. (2021) · INTERNLP · doi:10.18653/v1/2021.internlp-1.1 · Cited by 18 · cited 1× · see citing works
“In the realworld, however, humans often interact with an AI system for multiple rounds and ask clarification and follow-up questions to build a better mental model of the AI system (Zylberajch et al., 2021; Wu, 2022).”
BackgroundA call for more explainable AI in law enforcement
Matulionytė, R., & Hanif, A. (2021) · IEEE International Enterprise Distributed Object Computing Conference Workshops (EDOCW) · doi:10.1109/edocw52865.2021.00035 · Cited by 18 · cited 1× · see citing works
“…know when and how to use the model without misuse and overtrust, especially in high-stakes domains such as healthcare (Adadi and Berrada, 2020; Merry et al., 2021; Babic et al., 2021), legal (Deeks, 2019; Norkute et al., 2021), and law enforcement (Matulionyte and Hanif, 2021; Hall et al., 2022).”
BackgroundCan Explanations Be Useful for Calibrating Black Box Models?
Ye, X., & Durrett, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.429 · Cited by 17 · cited 2× · see citing works
“…an AI system is important, as it helps humans understand what an AI system can and cannot achieve (Chandrasekaran et al., 2018), which informs humans how to improve the system or appropriately deploy the system without misuse or overtrust (Cassidy, 2009; Bansal et al., 2019; Ye and Durrett, 2022).”
BackgroundLogical Satisfiability of Counterfactuals for Faithful Explanations in NLI
Sia, S., Belyy, A., Almahairi, A., et al. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2205.12469 · Cited by 17 · cited 1×
“While the concept of counterfactual simulatability has a long history (Doshi-Velez and Kim, 2017; Hase and Bansal, 2020; Sia et al., 2022), we are the first work to evaluate the counterfactual simulatability of free-form natural language explanations.”
BackgroundFRAME: Evaluating Simulatability Metrics for Free-Text Rationales
Chan, A., Nie, S.-L., Tan, L., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2207.00779 · Cited by 10
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 2×
“, 2022) and GPT-4 (OpenAI, 2023) to study how Dataset Generator BLEU Cos Jacc Sim.”
BackgroundAligning AI With Shared Human Values
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2008.02275 · cited 1×
“Building mental models also detects if the model biases against specific groups of people (Vig et al., 2020; Ravfogel et al., 2020) or encourages illegal behaviors against human values (Hendrycks et al., 2021; Bai et al., 2022).”
BackgroundSelf-citation
Flagged first, then by citations
8 of the 70 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. ArXiv.
- Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems.
- Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are reason- ers with self-verification. ArXiv.
- Tongshuang Wu. 2022. Interactive AI Model Debug- ging and Correction. Ph.D. thesis, University of Washington.
Cited by
Sorted by citations
Exploring the role of large language models in the scientific method: from hypothesis to discovery
Zhang, Y., Khan, S. A., Mahmud, A., Yang, H., Lavin, A., Levin, M. G., Frey, J. G., Dunnmon, J., Evans, J., Bundy, A., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00019-5 · Cited by 109 · see citing works
“Their explanations are often inconsistent with their behaviours, and we cannot use their explanations to predict their behaviours in counterfactual scenarios 181 .”
InfluentialTurpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · see citing works
“Evaluating counterfactual simu-latability in the general case involves manually inspecting model explanations and determining their implications for model behavior on coun-terfactual inputs, which can be expensive and subjective (Chen et al., 2023).”
BackgroundFuture Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33 · see citing works
LLMs are one-shot URL classifiers and explainers
Rashid, F., Ranaweera, N., Doyle, B., & Seneviratne, S. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.111004 · Cited by 30 · see citing works
Self-Consistency of Large Language Models under Ambiguity
Bartsch, H., Jørgensen, O. J., Rosati, D., Hoelscher-Obermaier, J., & Pfau, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.7 · Cited by 12 · see citing works
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“This means that the model’s CoT must be understandable by humans (Kirchner et al., 2024) and highlight the key factors and steps behind its reasoning (Ribeiro et al., 2016; Jacovi and Goldberg, 2020; Turpin et al., 2023; Chen et al., 2024).”
Self-citationResearch Agenda for Sociotechnical Approaches to AI SafetyPreprint
Curtis, S., Iyer, R., Kirk‐Giannini, C. D., Krakovna, V., Krueger, D., Lambert, N., Marnette, B., McKenzie, C., Michael, J., Miyazono, E., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5097286
- Authors
- Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., & McKeown, K.
- Affiliations
- Columbia University (US)
- University of California, Berkeley (US)
- New York University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.08678
- OpenAlex
- W4384648622
- Semantic Scholar
- 69f2ba0f33a54e01de32c616b64e85d5d7194067
- Cited by
- 5
- References
- 70
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. Department of Defense
- Office of the Director of National Intelligence
- Defense Advanced Research Projects Agency
- Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- natural language explanations, large language models, explainable AI, counterfactual reasoning, reward modeling, human approval, reinforcement learning from human feedback, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
