Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
Open accessAbstract
Recent studies indicate that when faced with explicit biases in prompts, models often omit mentioning these biases in their Chain-of-Thought (CoT) output, revealing that verbalized reasoning can give an incorrect picture of how models arrive at conclusions (unfaithfulness). In this work, we show that unfaithful CoT also occurs on naturally worded, non-adversarial prompts without adding artificial biases or editing model outputs. We find that when separately presented with the questions "Is X bigger than Y?" and "Is Y bigger than X?", models sometimes produce superficially coherent arguments to justify systematically answering Yes to both or No to both, despite the contradiction. We present preliminary evidence that this is due to models' implicit biases towards Yes or No, labeling this Implicit Post-Hoc Rationalization. Our results reveal rates up to 13% for production models, and while frontier models are more faithful, none are entirely so, including thinking models like DeepSeek R1 (0.37%) and Sonnet 3.7 with thinking (0.04%). We also investigate Unfaithful Illogical Shortcuts, where models use subtly illogical reasoning to make speculative answers to hard math problems seem rigorously proven. Our findings indicate that while CoT can be useful for assessing outputs, it is not a complete account of the internal process that produced the model's answer and should be used with caution in agentic or safety-critical settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A model may invent its own reasoning process, creating a false empirical or justificatory story.
“A second, more insidious failure arises when the model hallucinates its own chain of thought, producing either an empirical or justi-ficational fiction, or both.7”
From When a Human‐in‐the‐Loop Amplifies the Risk of Misalignment: Considerations for Research Ethics Oversight (2026) · cites this paper as [7]
Zero-shot chain-of-thought prompting did not yield any gains in the paper's preliminary experiments.
“We experimented with the zero-shot CoT approach (Kojima et al., 2022) in our preliminary experiments, but it did not yield any gains (Arcuschin et al., 2025).”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Arcuschin et al., 2025
Section 2.1 describes asking models to reason step by step, and Appendix B describes a few-shot prompt for one baseline model. Neither reports preliminary zero-shot CoT experiments or whether they yielded gains. Searches for “zero-shot,” “preliminary experiments,” and “gains” found no passage establishing the attributed result. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 24
- Atanasova, P., Camburu, O.-M., Lioma, C., Lukasiewicz, T., Simonsen, J. G., and Augenstein, I. Faithfulness tests for natural language explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pp. 283-294, Toronto, Canada, July 2023. Association for Computational Lin- guistics.
- Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., and Farhi, D. Monitoring reasoning models for misbehavior and the risks of promoting obfusca- tion. arXiv preprint arXiv:2503.11926, March 2025. URL https://openai.com/index/chain- of-thought-monitoring/. PDF available at https://cdn.openai.com/pdf/34f2ada6-870f- 4c26-9790-fd8def56387f/CoT_Monitoring.pdf as of 10th March 2025. · arXiv
- Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., and Mckeown, K. Do models explain themselves? Counterfactual simulatability of natural language explana- tions. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 7880-7904. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/ chen24bl.html.
- Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Deni- son, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S. R., Leike, J., Kaplan, J., and Perez, E. Reasoning models don't always say what they think, 2025. URL https://arxiv.org/abs/2505. 05410.
- Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., Sanyal, S., Welleck, S., Ren, X., Ettinger, A., Harchaoui, Z., and Choi, Y. Faith and fate: Limits of transformers on compositionality. In Advances in Neural Information Processing Systems, 2023.
- Emmons, S., Jenner, E., Elson, D. K., Saurous, R. A., Rajamanoharan, S., Chen, H., Shafkat, I., and Shah, R. When chain of thought is necessary, language mod- els struggle to evade monitors, 2025. URL https: //arxiv.org/abs/2507.05246.
- Gema, A. P., Hägele, A., Chen, R., Arditi, A., Goldman- Wetzler, J., Fraser-Taliente, K., Sleight, H., Petrini, L., Michael, J., Alex, B., Minervini, P., Chen, Y., Benton, J., and Perez, E. Inverse scaling in test-time compute. Transactions on Machine Learning Research, 2025. Google. Our next-generation model: Gemini 1.5, 2 2024. URL https://blog.google/innovation-and- ai/products/google-gemini-next-generation- model-february-2024/. Google. Start building with Gemini 2.5 Flash, 4 2025a. URL https://developers.googleblog.com/ en/start-building-with-gemini-25-flash/. Google. Gemini 2.5: Our newest Gemini model with thinking, 3 2025b. URL https://blog.google/ technology/google-deepmind/gemini-model- thinking-updates-march-2025/.
- Gurnee, W. and Tegmark, M. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024.
- Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. Training large language models to reason in a continuous latent space. In Conference on Language Modeling, 2025.
- Hase, P. and Potts, C. Counterfactual simulation training for chain-of-thought faithfulness, 2026. URL https: //arxiv.org/abs/2602.20710.
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
- Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021b.
- Jacovi, A. and Goldberg, Y. Towards faithfully interpretable NLP systems: How should we define and evaluate faith- fulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198- 4205. Association for Computational Linguistics, 2020.
- Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., Chen, M., Cooney, A., Dafoe, A., Dragan, A., Emmons, S., Evans, O., Farhi, D., Greenblatt, R., Hendrycks, D., Hobbhahn, M., Hubinger, E., Irving, G., Jenner, E., Kokotajlo, D., Krakovna, V., Legg, S., Lindner, D., Luan, D., M ądry, A., Michael, J., Nanda, N., Orr, D., Pachocki, J., Perez, E., Phuong, M., Roger, F., Saxe, J., Shlegeris, B., Soto, M., Steinberger, E., Wang, J., Zaremba, W., Baker, B., Shah, R., and Mikulik, V. Chain of thought monitorability: A new and fragile opportunity for AI safety, 2025. URL https://arxiv.org/abs/ 2507.11473.
- Lambell, N. J., Evans, J. S. B. T., and Handley, S. J. Be- lief bias, logical reasoning and presentation order on the syllogistic evaluation task. In Proceedings of the Twenty First Annual Conference of the Cognitive Science Society, pp. 282-288, 1999.
- Landis, J. R. and Koch, G. G. The measurement of observer agreement for categorical data. Biometrics, 33(1):159- 174, 1977. doi: 10.2307/2529310. · DOI · Paper page
- Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Deni- son, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCan- dlish, S., Kundu, S., Kadavath, S., Yang, S., Henighan, T., Maxwell, T., Telleen-Lawton, T., Hume, T., Hatfield- Dodds, Z., Kaplan, J., Brauner, J., Bowman, S. R., and Perez, E. Measuring faithfulness in chain-of-thought reasoning. ArXiv, abs/2307.13702, 2023. · arXiv
- Li, B., Zhou, B., Wang, F., Fu, X., Roth, D., and Chen, M. Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination? In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 1: Long Papers), pp. 7675-7688, Mexico City, Mexico, June 2024. Asso- ciation for Computational Linguistics. doi: 10.18653/ v1/2024.naacl-long.424. URL https://aclanthology. org/2024.naacl-long.424/. · DOI
- Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., and Callison-Burch, C. Faithful chain- of-thought reasoning. In Proceedings of the 13th Interna- tional Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 305-329, Nusa Dua, Bali, November 2023. Association for Computational Linguistics.
- Martín, M. and Valiña, M. D. Heuristics, biases and the psychology of reasoning: State of the art. Psychology, 14 (2):264-294, 2023.
- M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z., and Fan, Z. Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671. · arXiv
- Yee, E., Li, A., Tang, C., Jung, Y. H., Paturi, R., and Bergen, L. Dissociation of faithful and unfaithful rea- soning in LLMs, 2024. URL https://arxiv.org/abs/ 2405.15092.
- Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In Advances in Neural Information Processing Systems, 2025.
- Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, J. The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 10495-10516. Association for Computational Linguistics, 2025.
Cited by
Sorted by citations
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“Arcuschin et al. (2025) also finds CoTs of reasoning models to be unfaithful, and studies two specific kinds of unfaithfulness: restoration errors (models silently correct their errors without verbalization) and unfaithful shortcuts (models use illogical reasoning without verbalization).”
Taylor, E. (2026) · Ethics & Human Research · doi:10.1002/eahr.70024
LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring
Li, C., Phuong, M., & Siegel, N. Y. (2025) · arXiv · doi:10.48550/arxiv.2508.00943 · Cited by 1 · see citing works
“Previous work has evaluated the faithfulness of CoT (i.e., the extent that CoT accurately represents the models’ internal reasoning process) and revealed un-faithfulness across tasks (Turpin et al., 2023; Siegel et al., 2024; Lanham et al., 2023) and in the wild (Arcuschin et al., 2025).”
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.07952
Algorithmic Collusion by Large Language Models
Fish, S., Gonczarowski, Y. A., & Shorrer, R. I. (2024) · arXiv · doi:10.48550/arxiv.2404.00806 · Cited by 10 · see citing works
- Authors
- Arcuschin, I., Janiak, J., Krzyzanowski, R., Rajamanoharan, S., Nanda, N., & Conmy, A.
- Affiliations
- Poseidon (Norway) (NO)
- European Commission (BE)
- ORCID
- Arcuschin, Iván
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2503.08679
- OpenAlex
- W4416030400
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 5 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Institute for Catastrophic Loss Reduction
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- chain-of-thought reasoning, unfaithfulness, implicit bias, post hoc rationalization, mathematical problem solving, large language models, agentic systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
