The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Open accessAbstract
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, Huan Sun. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Chain-of-Thought (CoT) prompting remains effective even when the demonstrated rationales contain invalid reasoning, retaining over 80–90% of CoT performance under various metrics. The key factors for effective CoT are that rationales stay relevant to the query and that their steps are correctly ordered, rather than being logically valid.
Key findings
- Invalid reasoning in demonstrations still yields high performance: on GSM8K with text-davinci-002, CoT reaches 48.3 Inter. F1 and 48.5 Answer Acc., while invalid reasoning reaches 43.9 Inter. F1 and 39.5 Answer Acc.; on Bamboogle, CoT reaches 45.2 Inter. Recall and 45.2 Answer F1, while invalid reasoning reaches 44.4 Inter. Recall and 39.4 Answer F1.
- Relevance and coherence are key: the “no relevance” setting (both components irrelevant) underperforms standard prompting on GSM8K (11.0 Answer Acc. vs. 15.4 for STD), and “no coherence for language templates” drops to 28.3 Inter. F1 from CoT’s 48.3.
- For bridging objects, relevance matters more than coherence: incoherent bridging objects achieve 39.2 Inter. F1 on GSM8K, while irrelevant bridging objects achieve 26.2 Inter. F1.
- At the instance level, there are cases where CoT is wrong and invalid reasoning is correct: 62/196 for GSM8K and 6/20 for Bamboogle.
Methods
- The authors run ablation experiments on two multi-step reasoning tasks: arithmetic reasoning on GSM8K (800 of 1319 test examples sampled) and multi-hop factual QA on Bamboogle (all 125 test samples).
- Main backbone is InstructGPT-175B text-davinci-002 with greedy decoding; additional results are reported for text-davinci-003, PaLM, and Flan-PaLM in the appendix.
- Ablations alter CoT rationales by invalid reasoning, destroying relevance or coherence of bridging objects and language templates, and combined no-relevance/no-coherence settings; evaluation includes extrinsic answer accuracy/F1 and intrinsic Recall/F1 of bridging objects.
Limitations
- Stated: experiments cover only arithmetic reasoning and multi-hop QA; symbolic tasks are hard to ablate because their rationales are highly template-based with little variation.
- Stated: invalid reasoning rationales were manually written, and a more systematic treatment of invalid reasoning is left for future work.
- Stated: intrinsic evaluation relies on correctness of bridging objects and ground-truth annotations, which may not generalize and are costly to obtain; evident from the checklist: all results are reported with a single run due to limited budgets.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
33 Own result · 17 Own interpretation · 6 Not checked yet · 5 Own method · 2 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this paper, we show that CoT reasoning is possible even with invalid demonstrations-prompting with invalid reasoning steps can achieve over 80-90% of the performance obtained using CoT under various metrics, while still generating coherent lines of reasoning during inference.”
- Also stated in: § Invalid Reasoning · plain
“We find that valid reasoning for the demonstrations matters only a small portion to the performance of CoT-by providing rationales with invalid reasoning, LLMs can achieve over 80-90% of the performance of CoT under various metrics while performing logically sound and pertinent reasoning.”
- Also stated in: § Invalid Reasoning · plain
“by providing rationales with completely invalid reasoning steps, the LLM can still achieve over 80-90% of the performance of CoT under various metrics while generating coherent lines of reasoning towards the answer ( §4).”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Invalid Reasoning · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, under Inter.Recall/Inter.F1, i.e., intrinsic evaluation, which is arguably a more faithful measurement of the rationale quality ( §3.3), all LLMs we tested can retain over 90% of the performance achieved under CoT prompting.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By checking the generated rationales for the invalid reasoning setting, we find that overall they look indistinguishable from the rationales generated by CoT prompting.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the instance level, after omitting samples where both settings get the correct/wrong answer, there is a significant portion for the remaining ones (62/196 for GSM8K, 6/20 for Bamboogle) where CoT gets the wrong answer and the invalid reasoning setting gets the correct answer.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Providing incoherent bridging objects ( 2 ) achieves better performance than providing irrelevant bridging objects ( 3 ), especially on the more challenging GSM8K dataset (39.2 v.s.26.2 Inter.F1).”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance drop is also uniform across samples with different levels of difficulty.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while still generating coherent lines of reasoning during inference.”
- Also stated in: § Invalid Reasoning · plain
“while generating coherent lines of reasoning towards the answer ( §4).”
- Also stated in: § Invalid Reasoning · plain
“while performing logically sound and pertinent reasoning.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Invalid Reasoning · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Different from the coherence of bridging objects 2 , the coherence of language templates 4 matters a lot to the performance of CoT prompting.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further experiments show that other aspects of the rationales, such as being relevant to the query and correctly ordering the reasoning steps, are much more important for effective CoT reasoning.”
- Also stated in: § Invalid Reasoning · plain
“Through further examinations, we identify and formulate other aspects of a CoT rationale ( §5), and find that being relevant to the query and correctly ordering the reasoning steps are the key for the effectiveness of CoT prompting.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Invalid Reasoning · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The distributions of errors in both cases are highly similar.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plainn = 20- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 15
- Probably not a substantial claim. 6
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can be seen that most of the settings for this section ( 2 -7 ) have rather large performance drops from CoT, where the low-performing ones approach or even underperform standard prompting.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By examining the predicted rationales, we find that the LLM is indeed generating rationales with incoherent language templates (14 out of 20 examples), which negatively affects reasoning.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The no relevance setting 7 where both components of the rationale have no relevance achieves significantly poorer performance than other ablation settings, and even underperforms standard prompting (STD) where no rationale is given on GSM8K.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In almost all cases where the predicted final answer is correct, the rationales do reach the answer with valid and sound reasoning steps (as in CoT), drastically different from those in the given demonstrations; for cases where the final answer is wrong, the errors the LLM makes are also in the same types with the errors made under CoT prompting.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each step of a rationale still follows the previous steps.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The rationales still use information from the query; more specifically, they still start from bridging objects mentioned in the query, and the language templates are related to the query.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 4.2 Results & Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, providing invalid reasoning for the rationales ( 1 ) overall only marginally harms the performance, and even outperforms CoT on GSM8K under intrinsic evaluation.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > A.3 Additional Results & Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the remaining settings where we ablate the relevance/coherence ( 2 -7 ), the same trend can be observed on the challenging GSM8K dataset, e.g., the model still suffers a lot when providing rationales that are irrelevant or have incoherent language templates.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > A.3 Additional Results & Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the relatively easier Bamboogle dataset, the high model capacity indicated by its impressive performance has basically erased significant impacts from the ablations, with the only standing observation that the model still needs the rationales to be relevant to maintain its performance.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > A.3 Additional Results & Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not use whole equations since we observe that the LLM may express the mathematical equation in different ways, e.g., “5 plus 3 is 8”, “5 + 3 = 8”.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 2729
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Relevance and coherence are key for the performance of CoT prompting.”
- Also stated in: § PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows. · hedged
“This suggests that overall, relevance and coherence are key for the performance of CoT.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows. · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which indicates that it is important for the bridging objects to be relevant, but not as important to have them in the right order to guide the LLM along the reasoning process.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > PaLM and Flan-PaLM in Appendix A.3. We summarize the main findings in what follows.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our intrinsic evaluation of the generated rationales is based on the correctness of bridging objects, which, even though is a good indicator of the quality of language templates (Appendix A.2) in our experiments, may not be a good metric in general cases.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, most (if not all) tasks there are highly templatebased and hence the reasoning steps have little variations, both within each example and across different examples.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These edits only slightly affect the performance of CoT; we show our edited demonstration examples and include more details in Appendix A.1.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 3.1 Datasets & In-context Exemplars
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to limited budgets, we report all results with a single run.This is transparent from the paper.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“LLMs may over-rely on their prior knowledge and ignore important information in the context that are presumably rare in the pretraining distribution, including those that are crucial for specifying the task semantics”
- Also stated in: § A.3 Additional Results & Discussion · hedged
“LLMs may over-rely on their prior knowledge and ignore important information in the context, including those that are crucial for specifying the task semantics”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > 6 Discussion
What the sources say
- Not checked yetcites Jang et al., 2023
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § A.3 Additional Results & Discussion · hedged
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“It also relies on ground truth bridging objects, which are usually not available and costly to annotate.”
§ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
There is little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance.
“Despite its success, there is still little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance.”
The paper relies on this without support.
The two tasks used, arithmetic reasoning and multi-hop factual question answering, are representative of multi-step reasoning tasks.
“We experiment on two representative tasks involving multi-step reasoning: arithmetic reasoning & multi-hop factual question answering (QA).”
The paper relies on this without support.
Pretraining corpora are not always accessible.
“However, the pretraining corpora are not always accessible”
The paper relies on this without support.
Low-level statistics are usually not adequate when the topics of interest are abstract and high-level skills such as reasoning.
“low-level statistics are usually not adequate when the topics of interest are abstract and highlevel skills such as reasoning”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Self-consistency [66] and ensemble refinement [9] are strategies that sample multiple model answers to arrive at a more consistent response, improving model reliability [9].”
From The Opportunities and Risks of Large Language Models in Mental Health (2024) · cites this paper as [66]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755
Self-citationRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Self-citationTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Research works printed but not linked: 2
- Irving Copi, Carl Cohen, and Victor Rodych. 2016. In- troduction to logic. Routledge.
- Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gre- gory Valiant. 2022. What can transformers learn in- context? a case study of simple function classes. In Advances in Neural Information Processing Systems, volume 35, pages 30583-30598.
Cited by
Sorted by citations
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344
Mersha, M. A., Lâm, K. N., Wood, J., AlShami, A. K., & Kalita, J. K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 346
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230
Self-citationThe Opportunities and Risks of Large Language Models in Mental Health
Lawrence, H. R., Schneider, R. A., Rubin, S. B., Matarić, M. J., McDuff, D. J., & Jones Bell, M. (2024) · JMIR Mental Health · doi:10.2196/59479 · Cited by 235
Large Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230
Self-citationArtificial Intelligence for Materials Discovery, Development, and Optimization
Madika, B., Saha, A., Kang, C., Buyantogtokh, B., Agar, J., Wolverton, C. M., Voorhees, P. W., Littlewood, P. B., Kalinin, S. V., & Hong, S. (2025) · ACS Nano · doi:10.1021/acsnano.5c04200 · Cited by 209
Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 247
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Large Language Models are Better Reasoners with Self-Verification
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., & Zhao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.167 · Cited by 110
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262
Empirical Study of Zero-Shot NER with ChatGPT
Xie, T., Li, Q., Zhang, J., Zhang, Y., Liu, Z., & Wang, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.493 · Cited by 110
Efficient Prompting for LLM-Based Generative Internet of Things
Xiao, B., Kantarcı, B., Kang, J., Niyato, D. T., & Guizani, M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3470210 · Cited by 77
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 127
Self-citationChoi, W. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04832-x · Cited by 54
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
Using ChatGPT and Persuasive Technology for Personalized Recommendation Messages in Hotel Upselling
Remountakis, M., Kotis, K., Kourtzis, B., & Tsekouras, G. E. (2023) · Information · doi:10.3390/info14090504 · Cited by 66
He, J., Wang, L., Hu, Y., Liu, N., Liu, H., Xu, X., & Shen, H. T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01785 · Cited by 47
Lee, S., Sim, W., Shin, D., Seo, W., Park, J., Lee, S. C., Hwang, S., Kim, S., & Kim, S. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3712701 · Cited by 66
Ontology Generation Using Large Language Models
Lippolis, A. S., Saeedizade, M. J., Keskisärkkä, R., zuppiroli, S., Ceriani, M., Aldo Gangemi, A., Blomqvist, E., & Nuzzolese, A. G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94575-5_18 · Cited by 46
Active Learning Principles for In-Context Learning with Large Language Models
Margatina, K., Schick, T., Αλέτρας, Ν., & Dwivedi-Yu, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.334 · Cited by 72
Subramonyam, H., Thakkar, D., Ku, A., Dieber, J., & Sinha, A. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713166 · Cited by 61
Evaluating zero-shot multilingual aspect-based sentiment analysis with large language models
Wu, C., Ma, B., Zhang, Z., Deng, N., He, Y., & Xue, Y. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02711-z · Cited by 41
Symbol tuning improves in-context learning in language models
Wei, J., Hou, L., Lampinen, A. K., Chen, X., Huang, D. Y., Tay, Y., Chen, X., Lu, Y., Zhou, D., Ma, T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.61 · Cited by 80
R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’
Zhang, H., Diao, S., Lin, Y., Fung, Y., Lian, Q., Wang, X., Chen, Y., Ji, H., & Zhang, T. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.394 · Cited by 28
Latest developments of generative artificial intelligence and applications in ophthalmology
Feng, X., Xu, K., Luo, M., Chen, H., Yang, Y., He, Q., Song, C., Li, R., Wu, Y., Wang, H., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100090 · Cited by 33
Self-citationGu, Z., Jia, W., Piccardi, M., & Yu, P. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103078 · Cited by 27
Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
Ye, S., Hwang, H., Yang, S., Yun, H., Kim, Y., & Seo, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29909 · Cited by 59
Dr.ICL: Demonstration-Retrieved In-context Learning
Luo, M., Xu, X., Dai, Z., Pasupat, P., Kazemi, M., Baral, C. R., Imbrasaitė, V., & Zhao, V. Y. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0012 · Cited by 86
ContractTinker: LLM-Empowered Vulnerability Repair for Real-World Smart Contracts
Wang, C., Zhang, J., Gao, J., Xia, L., Guan, Z., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695349 · Cited by 21
Large language model powered knowledge graph construction for mental health exploration
Gao, S., Yu, K., Yang, Y., Yu, S., Shi, C., Wang, X., Tang, N., & Hongtu Zhu, B. (2025) · Nature Communications · doi:10.1038/s41467-025-62781-z · Cited by 29
Selective Demonstrations for Cross-domain Text-to-SQL
Chang, S., & Fosler‐Lussier, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.944 · Cited by 30
Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
Nguyen, T., Luo, L., Shiri, F., Phung, D., Li, Y.-F., Vu, T.-T., & Haffari, G. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.168 · Cited by 17
Ge, S., Sun, Y., Cui, Y., & Wei, D. (2024) · IEEE Access · doi:10.1109/access.2024.3494054 · Cited by 24
Chen, Z., Hao, J., Sun, H., Zhao, L., Li, J., Qian, Q., Peng, Q., Wang, X., Cong, S., Shen, L., et al. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104278 · Cited by 17
Self-citationPhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation
Xue, Q., Yin, X., Yang, B., & Gao, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01754 · Cited by 43
Dai, T., Wang, F., & Chen, Q. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.113647 · Cited by 13
The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions
Ouyang, S., Wang, S., Liu, Y., Zhong, M., Jiao, Y., Iter, D., Pryzant, R., Zhu, C., Ji, H., & Han, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.146 · Cited by 29
Towards Specification-Driven LLM-Based Generation of Embedded Automotive Software
Patil, M. S., Ung, G., & Nyberg, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75434-0_9 · Cited by 20
Suong, T. P. M., Nghiem, L. H., & Tran, T. K. (2025) · Enterprise Information Systems · doi:10.1080/17517575.2025.2520213 · Cited by 11
Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Lin, H., & Zhang, Y. (2025) · Social Science Computer Review · doi:10.1177/08944393251366243 · Cited by 22
TypeFly: Low-Latency Drone Planning With Large Language Models
Chen, G., Yu, X., Ling, N., & Zhong, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3561282 · Cited by 31
Jin, W., Su, D., Tao, T., Wang, X., Wang, N., & Zhao, B. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00038-x · Cited by 14
DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion
He, H., Yang, H., Tuo, Z., Zhou, Y., Wang, Q., Zhang, Y., Liu, Z., Huang, W., Chao, H., & Yin, J. P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3600149 · Cited by 35
HiPrompt: Few-Shot Biomedical Knowledge Fusion via Hierarchy-Oriented Prompting
Lu, J., Shen, J., Xiong, B., Ma, W., Staab, S., & Yang, C. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591997 · Cited by 14
Self-citationTeach Me How to Argue: A Survey on NLP Feedback Systems in Argumentation
Guerraoui, C., Reisert, P., Inoue, N., Mim, F. S., Singh, K., Choi, J., Robbani, I., Naito, S., Wang, W., & Inui, K. (2023) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/2023.argmining-1.3 · Cited by 9
Prompting Large Language Models for Aerial Navigation
Balcı, E., Sarıgül, M., & Ata, B. (2024) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk63289.2024.10773467 · Cited by 9
A Survey on Large Language Models for Mathematical Reasoning
Wang, P.-Y., Liu, T.-S., Wang, C., Li, Z., Wang, Y., Yan, S., Jia, C., Liu, X.-H., Chen, X., Xu, J., et al. (2025) · ACM Computing Surveys · doi:10.1145/3786333 · Cited by 9
Guo, Q., Guo, Y., & Zhao, J. (2024) · Neural computing & applications (Print) · doi:10.1007/s00521-024-09728-5 · Cited by 10
Explainable and Interactive LLMs-Augmented Depression Detection in Social Media
Qin, W., Chen, Z., Yang, X., Wang, L., Lan, Y., Ren, W., & Hong, R. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3591755 · Cited by 9
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
Ye, X., & Durrett, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.41 · Cited by 11
Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.05080 · Cited by 7
Self-citationBhattacharya, A., Vanherwegen, T., & Verbert, K. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728321 · Cited by 8
OD-Mind: An ocean drilling expert knowledge query system driven by knowledge graph
Yang, Y., Cao, W., Li, Y., Chang, R., Yang, G., Gan, C., & Wu, M. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.122135 · Cited by 7
A dataset for evaluating clinical research claims in large language models
Zhang, B., Bornet, A., Yazdani, A., Khlebnikov, P., Milutinovic, M., Rouhizadeh, H., Amini, P., & Teodoro, D. (2025) · Scientific Data · doi:10.1038/s41597-025-04417-x · Cited by 11
Application of Generative AI to Derive Insight from Supply Chain & Logistics Contracts
Singh, A. P., Jia, T., Nalagatla, V., Cunningham, B., & Siwani, T. (2024) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/222932-ms · Cited by 5
Qin, W., Suo, L., Li, L., & Yang, F. (2025) · Applied Sciences · doi:10.3390/app15126651 · Cited by 10
Tsai, C., Lin, Y.-C., Hou, J.-U., Tsai, S., Yeh, P.-C., & Kao, C. (2025) · Digital Health · doi:10.1177/20552076251357468 · Cited by 4
Salman, A. M., Alsiyat, Y., Creese, S., & Goldsmith, M. (2025) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw67616.2025.00046 · Cited by 4
Sun, Q., Cui, J., Chang, F., Sheng, L., & Gu, Z. (2025) · Journal of Engineering Design · doi:10.1080/09544828.2025.2588551 · Cited by 4
Chen, Z., Gao, Y., Gu, S., Yong, Z., Li, L., Wang, R., Lei, Q., & Zeng, S. (2026) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00140-3 · Cited by 4
Enhancing Mathematical Reasoning in GPT-J Through Topic-Aware Prompt Engineering
Sukherman, L., & Folajimi, Y. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733807 · Cited by 4
Retrieval-Augmented Multi-Modal Chain-of-Thoughts Reasoning for Large Language Models
Liu, B., Lyu, C., Min, Z., Wang, Z., Su, J., & Wang, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228391 · Cited by 4
Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review
Habib, A., Abdulmahmod, O. F., Raza, M., Gu, Y. H., Aydoğan, M., & Al-antari, M. A. (2025) · 2025 9th International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap68205.2025.11222281 · Cited by 3
Formal Verification of Business Constraints in Workflow-Based Applications
Stoica, F., & Stoica, L. F. (2024) · Information · doi:10.3390/info15120778 · Cited by 3
Liao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
Generative AI Supporting Project Management: A Vertical Integration Using OpenAI MyGPT
Felicetti, A. M., Rao, D., & Ammirato, S. (2025) · Future of business and finance · doi:10.1007/978-3-031-84017-3_6 · Cited by 3
DyBooster: Leveraging large language model as booster for dynamic recommendation
Tang, H., Sun, X., Wu, S., Cui, Z., Xu, G., & Li, Q. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128080 · Cited by 4
Ahmad, P. N., Shah, A. M., & Lee, K. Y. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02712-6 · Cited by 4
Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces
Zhou, R., Nguyen, G., Kharya, N., Nguyen, A., & Agarwal, C. (2026) · Proceedings of the 31st International Conference on Intelligent User Interfaces · doi:10.1145/3742413.3789134 · Cited by 2
Kou, J., Lu, X., Yin, A., Zhang, W., Fang, X., Li, Y., Pan, T., Zhang, L., Cheng, G., Yuan, H., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189154 · Cited by 2
Generating Feedback for School Students Essay with Large Language Models
Zhang, D., Hoang, T. N., Zhu, Y., Wang, R., & Crouch, P. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3055-7_25 · Cited by 2
Multi-Dimensional Evaluation of Auto-Generated Chain-of-Thought Traces in Reasoning Models
Becerra-Monsalve, L. F., Sanchez-Torres, G., & Bedoya, J. B. B. (2026) · Applied Informatics · doi:10.3390/ai7010035 · Cited by 3
Route Optimization Reimagined: Multi-Modal Large Language Models for Next-Generation Vehicle Routing
Albalkhi, S. Y., Alotaibi, D. F., Dimitriou, T. D., & Ahmad, I. (2026) · IEEE Access · doi:10.1109/access.2026.3663141 · Cited by 2
EveMRC: Two-Stage Bidirectional Evidence Modeling for Multi-Choice Machine Reading Comprehension
Xu, H., Chen, L., Sun, L., Cao, R., Ma, D., & Yu, K. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3542294 · Cited by 1
Role-Specific Reward Design with Large Language Model for StarCraft II
Li, S., Lou, H., Zhang, X., Zeng, X., Shen, Z., & Li, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890857 · Cited by 2
Zhang, J., Li, X., Fang, J., Wei, W., & Cui, M. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4298-4_3 · Cited by 1
Double-Feedback: Enhancing Large Language Models Reasoning in Robotic Tasks by Knowledge Graphs
Wang, H., Zhang, S., Wang, S., Jiang, T., & Ge, Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3562776 · Cited by 3
Lu, B., Wang, X., Ming, W., & Wang, X. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01290-8 · Cited by 1
Prompt engineering for generating high-quality conversations and its evaluation
Yamazaki, H., Asai, T., & Ando, K. (2025) · Nonlinear Theory and Its Applications IEICE · doi:10.1587/nolta.16.909 · Cited by 2
Gonçalves, J. C., & Maia, M. D. A. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.9964 · Cited by 1
Wang, C., Cao, Z. G., & Zhou, K. (2025) · ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) · doi:10.1109/snpd65828.2025.11253421 · Cited by 2
Kwon, H., Abbas, L., Kang, J., Lu, L., & McLeod, D. M. (2026) · Data & Policy · doi:10.1017/dap.2025.10052 · Cited by 1
Yoo, Y., Kim, S., & Liu, J. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104898 · Cited by 1
Nguyen, Q. M., Chen, F., Liu, C., Campbell, I. M., Zhang, G., Wu, D., Szigety, K. M., Sheppard, S. E., Ahimaz, P. R., Ta, C., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02733-z · Cited by 1
Automated Comment Generation Based on the Large Language Model
Cai, K., Zhou, J., Kong, L., Liang, D., & Li, X. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0730-0_25 · Cited by 1
Kulikov, V., Neychev, R., & Makarov, I. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-67008-4_4 · Cited by 1
Multi-Hop Arabic LLM Reasoning in Complex QA
Abdelazim, H., Begemy, T., Galal, A., Sedki, H., & Mohamed, A. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.10.179 · Cited by 1
Post-Incorporating Code Structural Knowledge Into Pretrained Models via ICL for Code Translation
Du, Y., Sun, H., & Li, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3605768
Self-citationEvaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
Kawabata, A., & Sugawara, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.9 · Cited by 3
A dataset for evaluating clinical research claims in large language modelsPreprint
Zhang, B., Yazdani, A., Bornet, A., Khlebnikov, P., Milutinovic, M., Rouhizadeh, H., Amini, P., & Teodoro, D. (2024) · medRxiv · doi:10.1101/2024.10.08.24315103
AI as the Interpreter: Deciphering Mental Health Stories in the Social Media Realm (Preprint)Preprint
Saima, R., Ahmed, K. E. U., & O’Connor, M. (2025) · doi:10.2196/preprints.71219
A Non-Ergodic Framework for Understanding Emergent Capabilities in Large Language ModelsPreprint
Marin, J. (2025) · Qeios · doi:10.32388/00jqn6
Preliminary Study of LLM-Based Wordlist Generation for Validating Broken Web Access Control
Ng, K. K. S., Yan, F., & Hung, K. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902771
Zanina, A., & Gröner, C. U. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93418-6_17
Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pretrained Priors
Yang, H., Yang, Z., Zhang, Y., Zhu, S., & Yang, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_34
Differential diagnosis making with large language models and probabilistic logic program
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30046-2.00010-7
Metadata
- Authors
- Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H.
- Affiliations
- The Ohio State University (US)
- University of Washington (US)
- Google (United States) (US)
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 2717-2739
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4385571045
- Cited by
- 120
- References
- 26
- Citations by year
- 2026: 23 · 2025: 57 · 2024: 22 · 2023: 18
- FWCI
- 21.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1942980, 2112606, 1815674, CAREER)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- chain-of-thought prompting, large language models, prompting strategies, LLM reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
