A comparison of chain-of-thought reasoning strategies across datasets and models
Open accessAbstract
Emergent chain-of-thought (CoT) reasoning capabilities promise to improve the performance and explainability of large language models (LLMs). However, uncertainties remain about how reasoning strategies formulated for previous model generations generalize to new model generations and different datasets. In this small-scale study, we compare different reasoning strategies induced by zero-shot prompting across six recently released LLMs (davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl and Cohere command-xlarge). We test them on six question-answering datasets that require real-world knowledge application and logical verbal reasoning, including datasets from scientific and medical domains. Our findings demonstrate that while some variations in effectiveness occur, gains from CoT reasoning strategies remain robust across different models and datasets. GPT-4 benefits the most from current state-of-the-art reasoning strategies and performs best by applying a prompt previously discovered through automated discovery.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Chain-of-Thought prompting and natural language explanation generation produce outputs that align with clinical reasoning conventions, mimicking the differential diagnosis process familiar to practitioners.
“Chain-of-Thought prompting [62,64,78] and natural language explanation generation [72,74,75] produce outputs that align with clinical reasoning conventions, mimicking the differential diagnosis process familiar to practitioners.”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [64]
The Methods (Prompts; page unavailable) describe chain-of-thought prompts that elicit step-by-step reasoning, and the Methods (Datasets; page unavailable) include medical question-answering tasks. But the paper does not establish that the resulting outputs align with clinical reasoning conventions or mimic practitioners’ differential-diagnosis process; the Discussion (Future work; page unavailable) says evaluations of reasoning-chain quality and explanatory utility still need to be conducted. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 2× · see citing works
“We acknowledge that prompts which are semantically similar but differ syntactically can lead to varied outcomes, as observed in medical datasets ( Liévin, Hother & Winther, 2022 ).”
BackgroundResultHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · see citing works
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.5 · Cited by 70 · see citing works
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Ling Z Fang Y Li X Huang Z Lee M Memisevic R Su H 2024 Deductive verification of chain-of-thought reasoning Advances in Neural Information Processing Systems 36 36407 36433
- Ott S Hebenstreit K Liévin V Hother CE Moradi M Mayrhauser M Praas R Winther O Samwald M 2023 ThoughtSource: a central hub for large language model reasoning data 2301.11596 PMC10409727 37553439
- Pal A Umapathi LK Sankarasubbu M 2022 Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering Conference on health, inference, and learning PMLR 248 260
- Saunders W Yeh C Wu J Bills S Ouyang L Ward J Leike J 2022 Self-critiquing models for assisting human evaluators 2206.05802
- Talmor A Herzig J Lourie N Berant J 2019 CommonsenseQA: a question answering challenge targeting commonsense knowledge https://aclanthology.org/N19-1421/
- Xie Z Thiem S Martin J Wainwright E Marmorstein S Jansen P 2020 Worldtree v2: a corpus of science-domain structured explanations and inference patterns supporting multi-hop inference Proceedings of the twelfth language resources and evaluation conference 5456 5473
- Zhou Y Muresanu AI Han Z Paster K Pitis S Chan H Ba J 2023b Large language models are human-level prompt engineers The eleventh international conference on learning representations
- Zhou X Zhang Y Cui L Huang D 2020 Evaluating commonsense in pre-trained language models Proceedings of the AAAI conference on artificial intelligence 34 05 9733 9740 10.1609/aaai.v34i05.6523 · DOI
Cited by
Sorted by citations
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
- Authors
- Hebenstreit, K., Praas, R., Kiesewetter, L. P., & Samwald, M.
- Affiliations
- Medical University of Vienna (AT)
- KTH Royal Institute of Technology (SE)
- Humboldt-Universität zu Berlin (DE)
- Medical University of Vienna, Center for Medical Data Science, Institute of Artificial Intelligence, Vienna, Austria
- Venue
- PeerJ Computer Science
- Publisher
- PeerJ
- Published
- 2024-04-30
- Volume
- 10
- Pages
- e1999
- ISSN
- 2376-5992 (ISSN-L)
- PMC
- PMC11157560
- OpenAlex
- W4396519055
- Cited by
- 15
- References
- 10
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 1
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PeerJ Computer Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- chain-of-thought reasoning, large language models, zero-shot prompting, reasoning strategies, question answering, GPT-4, GPT-3.5-turbo, Flan-T5, cross-model generalization, cross-dataset generalization, scientific reasoning, medical reasoning, logical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
