The DOI matches its Crossref record; 496 of the 496 references it cites were found in a registry or an index; 110 of the 234 sources its claims cite were checked for support.
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
Open accessAbstract
The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), fueling a paradigm shift in information acquisition. Nevertheless, LLMs are prone to hallucination, generating plausible yet nonfactual content. This phenomenon raises significant concerns over the reliability of LLMs in real-world information retrieval (IR) systems and has attracted intensive research to detect and mitigate such hallucinations. Given the open-ended general-purpose attributes inherent to LLMs, LLM hallucinations present distinct challenges that diverge from prior task-specific models. This divergence highlights the urgency for a nuanced understanding and comprehensive overview of recent advances in LLM hallucinations. In this survey, we begin with an innovative taxonomy of hallucination in the era of LLM and then delve into the factors contributing to hallucinations. Subsequently, we present a thorough overview of hallucination detection methods and benchmarks. Our discussion then transfers to representative methodologies for mitigating LLM hallucinations. Additionally, we delve into the current limitations faced by retrieval-augmented LLMs in combating hallucinations, offering insights for developing more robust IR systems. Finally, we highlight the promising research directions on LLM hallucinations, including hallucination in large vision-language models and understanding of knowledge boundaries in LLM hallucinations.
Overview
CoChat Summary
TL;DR
This survey proposes a taxonomy of LLM hallucination into factuality and faithfulness, then reviews causes, detection methods, benchmarks, mitigation strategies, and RAG-specific challenges. It also outlines open directions including hallucinations in large vision-language models and understanding knowledge boundaries in LLMs.
Key findings
- Redefines LLM hallucination as factuality hallucination (factual contradiction: entity-error, relation-error; factual fabrication: unverifiability, overclaim) and faithfulness hallucination (instruction inconsistency, context inconsistency, logical inconsistency).
- Identifies causes across data (misinformation/biases, knowledge boundary, inferior alignment data), training (pre-training, SFT, RLHF), and inference (imperfect decoding, over-confidence, softmax bottleneck, reasoning failure).
- Reviews detection and benchmarks: factuality detection via fact-checking and uncertainty estimation; faithfulness detection via fact-based, classifier-based, QA-based, uncertainty-based, and LLM-based metrics. Reported benchmark sizes include TruthfulQA (817 questions), HaluEval (30,000 task-specific and 5,000 general samples), FreshQA (600 hand-crafted questions), and HaluEval 2.0 (8,770 questions across five domains).
- Mitigation is organized by cause: data filtering, model editing, RAG; pre-training/alignment fixes; and factuality/faithfulness enhanced decoding. RAG still faces retrieval failure and generation bottleneck.
Methods
- Design: a survey/overview that proposes a taxonomy and organizes literature on LLM hallucination principles, causes, detection, benchmarks, mitigation, RAG challenges, and future directions.
- Data or sample: no primary dataset or new experiments; synthesizes existing research and benchmarks such as TruthfulQA, HaluEval, FreshQA, and HaluEval 2.0.
- Main analysis: conceptual categorization and qualitative synthesis; no quantitative meta-analysis is reported.
Limitations
- The paper does not explicitly state limitations of the survey itself.
- It states limitations of reviewed methods: RAG can be impacted by irrelevant retrievals and shallow retriever-generator interactions; fact-checking solely via LLM parametric knowledge may yield inaccurate assessments; classifier-based faithfulness metrics are constrained by scarcity of annotated data.
- Text makes evident: as a survey, it presents no original empirical validation or new benchmark results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
76 Valid · 5 Not valid · 48 Not checked · 108 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent studies have highlighted a strong correlation between the model's accuracy on general domain questions and the volume of relevant documents [145] or entity popularity [204] within the pre-training corpora.”
What the sources say
- Supportedcites [145] [145] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large Language Models Struggle to Learn Long-Tail Knowledge. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (Proceedings of Machine Learning Research, Vol. 202) , Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 15696-15707. https: //proceedings.mlr.press/v202/kandpal23a.html - source
The citation to this paper is attached to the volume of relevant documents, not the separate entity-popularity claim. Section 3.1 (p. 5) reports a strong correlation between question-answering accuracy and relevant pre-training document count across the tested models. (LLM, conf=0.94)
- Partly supportedcites [204] [204] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 9802-9822. https://doi.org/10.18653/v1/2023.acl-long.546 - sourceExperimental — accuracy not yet measured
The “Subject entity popularity predicts memorization” section reports a positive correlation between entity popularity and question-answering accuracy for almost all relationship types. However, the paper measures popularity using Wikipedia page views as a proxy, not entity frequency within pre-training corpora. Its Limitations section also says the experiments rely on synthetic datasets and that applicability to naturally occurring questions is not firmly established. (LLM, conf=0.87)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Such biases can be inadvertently acquired from internet-based texts, which are rife with diverse and biased viewpoints, and subsequently be propagated into the generated content [155].”
What the sources say
- Partly supportedcites [155] [155] Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. 2023. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, Dubrovnik, Croatia, 3206-3219. https://aclanthology.org/2023.eacl-main.234 - sourceExperimental — accuracy not yet measured
The Abstract and §6 Discussion report that pre-training biases can propagate into generated summaries as hallucinated facts. But §6 leaves the source of those biases for future work and treats stereotyping in pre-training data as one possibility; it does not establish that they were acquired from internet-based texts. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Neural networks possess an intrinsic tendency to memorize training data [35]”
What the sources say
- Partly supportedcites [35] [35] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) . 2633-2650. - sourceExperimental — accuracy not yet measured
Section 3.1 (p. 3) says memorization is in many ways an essential component of language models, and the Introduction (p. 1) demonstrates that large language models memorize individual training examples. The claim extends that finding to neural networks generally; the cited passages do not establish an intrinsic tendency across all neural networks. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, certain biases are intrinsically tied to hallucinations, especially those related to gender [242] and nationality [155, 226].”
What the sources say
- Not matchedcites [242]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [155] [155] Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. 2023. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, Dubrovnik, Croatia, 3206-3219. https://aclanthology.org/2023.eacl-main.234 - source
The Introduction and Section 3.3 report a strong association between pre-trained models’ name–nationality bias and nationality hallucinations in downstream text summarization. The citation to this paper is attached to nationality; the separate gender citation is not assessed here. (LLM, conf=0.89)
- Not matchedcites [226]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“this memorization tendency grows with model size [34, 54]”
What the sources say
- Supportedcites [34] [34] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. ArXiv preprint abs/2202.07646 (2022). https://arxiv.org/ abs/2202.07646 - source
Section 4.1 (p. 4) reports that larger models memorize significantly more than smaller models. The Introduction (p. 2) specifies that this finding is within model families, and the Conclusion (p. 9) describes memorization as scaling with model size. (LLM, conf=0.94)
- Supportedcites [54] [54] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, H source
Section 7, “Memorization” (p. 35), explicitly concludes that larger models have a higher memorization rate than smaller models. It also reports that the exact-continuation match rate rises from 1.6% for the 8B model to 2.4% for the 540B model. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
102 more claims · 41 with their cited sources checked
Create a free account to see all 237 claims and to verify the 13 references still open.
- Probably not a substantial claim. 97
- It may state two separate claims as one. 55
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, LLMs exhibit hallucination and fabrication errors that are unacceptable for quantitative modeling [19, 20, 21].”
From Structured Schemas for Provenance‐Rich, LLM ‐Assisted QSP Model Calibration (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In particular, the generation of non-existent academic references or legal precedents has been widely recognized as a critical issue [4].”
From Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Retrieval augmentation, tool use, citation systems, constrained decoding, domain-specific training, and multimodal grounding can all supply partial external anchoring, and they alter both the frequency and the detectability of unsupported claims [15,29].”
From Hallucination as epistemic pollution: epistemic injustice and credibility in futures-oriented science education (2026) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These measures do reduce the frequency of unsupported claims and improve the odds that a learner can detect one, and the argument here does not depend on denying it [15,29].”
From Hallucination as epistemic pollution: epistemic injustice and credibility in futures-oriented science education (2026) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Retrieval-augmented generation evaluation checks whether model claims are factually grounded in retrieved documents.It does not check whether retrieved evidence is attributed to the correct entity.”
From Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (2026) · cites this paper as Huang et al., 2025
Experimental — accuracy not yet measured
21 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.3 Hallucinations in Large Language Models · p. 7
“the final answer of x=4 is inconsistent with the reasoning chain, leading to an incorrect result.”
§ 2.3 Hallucinations in Large Language Models · p. 6
“Step 2: Divide both sides by 2 to get: x = 3. So the result of the equation is x=3.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 1×
“To align with human preferences, RLHF optimizes the LLM to generate outputs that maximize the reward provided by the trained preference model, typically employing a reinforcement learning algorithm, such as Proximal Policy Optimization (PPO)(Schulman et al., 2017).”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“While scaling up pre-training data substantially enhances the competencies of LLMs (Kaplan et al., 2020; Hoffmann et al., 2022), challenges arise in maintaining consistent data quality, which can potentially introduce misinformation and biases (Bender et al., 2021; Weidinger et al., 2021).”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 10×
“By retrieving evidence from external knowledge bases, RAG enables LLMs to be equipped with up-to-date knowledge and generate responses conditioning on relevant evidence.”
InfluentialLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“While scaling up pre-training data substantially enhances the competencies of LLMs (Kaplan et al., 2020; Hoffmann et al., 2022), challenges arise in maintaining consistent data quality, which can potentially introduce misinformation and biases (Bender et al., 2021; Weidinger et al., 2021).”
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Proceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 6×
“The current definition of hallucinations aligns with prior research (Ji et al., 2023a), characterizing them as generated content that is nonsensical or unfaithful to the provided source content.”
InfluentialLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · cited 1×
“Typically, RAG follows a retrieve-then-read pipeline, where relevant contextual documents are firstly retrieved by a retriever (Karpukhin et al., 2020) from external sources, and then the desired output is generated by a generator conditioning on both input text and retrieved documents.”
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
Suffix Arrays: A New Method for On-Line String Searches
Manber, U., & Myers, G. (1993) · SIAM Journal on Computing · doi:10.1137/0222058 · Cited by 1,915 · cited 1×
“In addition, a more efficient method utilizes the construction of a suffix array (Manber and Myers, 1993), enabling effective computation of numerous substring queries in linear time.”
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 2×
“While significant efforts have been undertaken in exploring their knowledge storage (Geva et al., 2021; Meng et al., 2022)and probing (Petroni et al., 2019; Zhong et al., 2021; Yu et al., 2023c), the exact mechanism by which LLMs capture the factual knowledge remains elusive.”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 1×
“However, when confronted with intricate challenges like multi-step reasoning (Yang et al., 2018c) and long-form question answering (Fan et al., 2019; Stelmakh et al., 2022), traditional one-time retrieval may fall short.”
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
On the resemblance and containment of documents
Broder, A. Z. (2002) · Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171) · doi:10.1109/sequen.1997.666900 · Cited by 1,698
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS
BRADLEY, R. A., & TERRY, M. E. (1952) · Biometrika · doi:10.1093/biomet/39.3-4.324 · Cited by 1,434
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 1×
“Stochastic sampling (Fan et al., 2018; Holtzman et al., 2020) is currently the prevailing decoding strategy employed by these LLMs.”
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · cited 1×
“However, as demonstrated in prior studies (Voita et al., 2019; Beltagy et al., 2020; Liu et al., 2023e), language models often exhibit a localized focus within their attention mechanisms, giving priority to nearby words and resulting in a notable deficit in context attention (Shi et al., 2023b).”
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Self-citationG-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
Proceedings of the 29th ACM International Conference on Multimedia
(2021) · doi:10.1145/3474085 · Cited by 922
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
Contrastive Decoding: Open-ended Text Generation as Optimization
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J. M., Hashimoto, T., Zettlemoyer, L., & Lewis, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.687 · Cited by 869
Textbooks Are All You Need II: phi-1.5 technical reportPreprint
Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., & Lee, Y. T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05463 · Cited by 850
Can AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 810
Contrastive Decoding: Open-ended Text Generation as OptimizationPreprint
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T. B., Zettlemoyer, L., & Lewis, M. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2210.15097 · Cited by 806
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · cited 1×
“…language understanding and generation capabilities, Large Vision-Language Models (LVLMs) have exhibited remarkable vision-language capabilities(Zhu et al., 2023a; Liu et al., 2023d; Yu et al., 2021; Huang et al., 2023b; Maaz et al., 2023; Chen et al., 2023e; Yu et al., 2023b; Zellers et al., 2019).”
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01274 · Cited by 728
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · cited 2×
“…detection benchmarks, most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al.,…”
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction TuningPreprint
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.32487.42402 · Cited by 644
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023) · arXiv · doi:10.48550/arxiv.2306.13063 · Cited by 618
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · cited 2×
“Unlike previous studies that predominantly focused on specific tasks such as summarization (Fabbri et al., 2021; Tang et al., 2022)or particular domains such as world knowledge (Miao et al., 2023), this benchmark assesses factuality across five domains: world knowledge, science and technology,…”
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575 · cited 3×
“Shuster et al. (2021) introduced the Knowledge F1 metric to assess how well the model’s generation aligns with the supplied knowledge.”
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Evaluating Object Hallucination in Large Vision-Language Models
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.20 · Cited by 559
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523
Woodpecker: hallucination correction for multimodal large language models
Yin, S., Fu, C., Zhao, S., Xu, T. B., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., & Chen, E. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4251-x · Cited by 518
Self-citationC-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2023) · arXiv · doi:10.48550/arxiv.2309.07597 · Cited by 514
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512 · cited 1×
“As a result, when confronted with problems necessitating domain-specific knowledge, such as medical (Li et al., 2023g; Singhal et al., 2023) and legal (Yu et al., 2022; Katz et al., 2023) questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication.”
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 512
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 505
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination EvaluationPreprint
Wang, J., Wang, Y., Xu, G., Zhang, J., Gu, Y., Jia, H., Wang, J., Xu, H., Yan, M., Zhang, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07397 · Cited by 475
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
Reasoning with Language Model Prompting: A SurveyPreprint
Qiao, S.-F., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S.-M., Tan, C.-Q., Huang, F., & Chen, H.-J. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09597 · Cited by 459
In-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449
GPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2302.04166 · Cited by 446
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
Ovadia, O., Brief, M., Mishaeli, M., & Elisha, O. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.15 · Cited by 443
SemDeDup: Data-efficient learning at web-scale through semantic deduplicationPreprint
Abbas, A., Tirumala, K., Simig, D., Ganguli, S., & Morcos, A. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09540 · Cited by 439
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
Maaz, M., Rasheed, H., Khan, S., & Khan, F. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.679 · Cited by 424
Extracting Latent Steering Vectors from Pretrained Language Models
Subramani, N., Suresh, N., & Peters, M. E. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.48 · Cited by 423
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2023) · doi:10.2139/ssrn.4389233 · Cited by 422
LLM Lies: Hallucinations are not Bugs, but Features as Adversarial ExamplesPreprint
Yao, J.-Y., Ning, K.-P., Liu, Z., Ning, M., Li, Y., & Yuan, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01469 · Cited by 417
Benchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2301.13848 · Cited by 415
Benchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00632 · Cited by 415
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 404
A Comprehensive Study of Knowledge Editing for Large Language ModelsPreprint
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.01286 · Cited by 404
Survey on Factuality in Large Language Models
Wang, C., Liu, X., Yue, Y., Guo, Q., Hu, X., Tang, X., Zhang, T., Cheng, J., Yao, Y., Hu, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3742420 · Cited by 396
Analyzing and Mitigating Object Hallucination in Large Vision-Language ModelsPreprint
Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., & Yao, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.00754 · Cited by 391
Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., & Liu, T. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.65 · Cited by 386
Self-citationC-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08322 · Cited by 373
RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective AugmentationPreprint
Xu, F.-Y., Shi, W., & Choi, E. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.04408 · Cited by 369
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356 · cited 2×
“Notable implementations include (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021).”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.557 · Cited by 349
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · arXiv · doi:10.48550/arxiv.2212.10509 · Cited by 349
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2305.11747 · Cited by 340
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340 · cited 1×
“• Long-tail Knowledge Within the vast knowledge landscape that LLMs draw upon, a notable challenge emerges in the form of long-tail knowledge utilization (Kandpal et al., 2023; Mallen et al., 2023).”
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · arXiv · doi:10.48550/arxiv.2305.01937 · Cited by 331
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · cited 1×
“However, when confronted with intricate challenges like multi-step reasoning (Yang et al., 2018c) and long-form question answering (Fan et al., 2019; Stelmakh et al., 2022), traditional one-time retrieval may fall short.”
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306 · cited 1×
“Notable implementations include (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021).”
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citationDeduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304 · cited 1×
“However, the inherent memo-rization capability becomes problematic in the context of duplicated information present within pre-training data (Lee et al., 2022a; Kandpal et al., 2023; Paullada et al., 2021).”
RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on AgriculturePreprint
Balaguer, A. D. L., Benara, V., Cunha, R. L. F., Filho, R. D. M. E., Hendry, T., Holstein, D., Marsman, J., Mecklenburg, N., Malvar, S., Nunes, L. O., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.08406 · Cited by 304
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303
Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.389 · Cited by 299
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 1×
“A burgeoning line of research (Khot et al., 2022; Yao et al., 2022; Press et al., 2022; He et al., 2023; Trivedi et al., 2023) endeavors to tackle such intricate tasks by decomposing them into more manageable sub-tasks.”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
Varshney, N., Yao, W., Zhang, H., Chen, J., & Yu, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.03987 · Cited by 293
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
Yu, W., Zhang, H., Pan, X., Cao, P., Ma, K., Li, J., Wang, H., & Yu, D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.813 · Cited by 289
Self-citationLIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11206 · Cited by 275
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2400 · Cited by 275
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265 · cited 1×
“In line with this, numerous studies (Falke et al., 2019; Maynez et al., 2020) have trained classifiers on NLI datasets to identify factual inaccuracies, especially in the context of abstract summarization.”
Query Rewriting in Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.322 · Cited by 263
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262
Context-faithful Prompting for Large Language Models
Zhou, W., Zhang, S., Poon, H., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.968 · Cited by 261
ToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.89 · Cited by 252 · cited 1×
“…such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al., 2022), data-to-text(Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al., 2021).”
ToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A. P., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · arXiv · doi:10.48550/arxiv.2004.14373 · Cited by 252 · cited 1×
“…such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al., 2022), data-to-text(Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al., 2021).”
Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language ModelsPreprint
Yu, W., Zhang, H.-M., Pan, X., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2311.09210 · Cited by 251
Self-citationSelf-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 251
Rohde, H., Elsner, M., & Clarke, A. V. (2014) · The Association for Computational Linguistics · Cited by 248
Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
Paul, D., West, R., Bosselut, A., & Faltings, B. V. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.882 · Cited by 242
Instruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.10792 · Cited by 241
Entity-level Factual Consistency of Abstractive Text Summarization
Nan, F., Nallapati, R. M., Wang, Z., dos Santos, C. N., Zhu, H., Zhang, D., McKeown, K. R., & Xiang, B. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.235 · Cited by 238 · cited 1×
“Nan et al. (2021) introduced a metric to quantify the extent of entity hallucination, which calculates the precision of named-entities in the summary against the source entities.”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238 · cited 1×
“…have proposed methods such as fine-tuning on adversarial datasets (Barrantes et al., 2020), decomposing the entailment decisions at the dependency arc level (Goyal and Durrett, 2020), and segmenting documents into sentence units then aggregating scores between sentence pairs (Laban et al., 2022).”
The power of noise: Redefining retrieval for rag systems
Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., & Silvestri, F. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657834 · Cited by 236
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence
Wu, K., Wu, E., & Zou, J. (2024) · neural information processing systems · doi:10.52202/079017-1053 · Cited by 234
Large Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225
Natural Language Reasoning, A Survey
Yu, F., Zhang, H., Tiwari, P., & Wang, B. (2024) · ACM Computing Surveys · doi:10.1145/3664194 · Cited by 224
Faithful Logical Reasoning via Symbolic Chain-of-ThoughtPreprint
Xu, J., Fei, H., Pan, L., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2405.18357 · Cited by 217
Query2doc: Query Expansion with Large Language Models
Wang, L., Yang, N., & Wei, F. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.585 · Cited by 214
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., & Grave, E. (2022) · arXiv · doi:10.48550/arxiv.2208.03299 · Cited by 213
Inspecting and Editing Knowledge Representations in Language ModelsPreprint
Hernandez, E. J., Li, B. Z., & Andreas, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.00740 · Cited by 213
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
REPLUG: Retrieval-Augmented Black-Box Language Models
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.463 · Cited by 209
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204
If beam search is the answer, what was the question?
Meister, C., Cotterell, R., & Vieira, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.170 · Cited by 203 · cited 1×
“The rationale for incorporating randomness into decoding strategies stems from the realization that high likelihood sequences often result in surprisingly low-quality text, which is called likelihood trap (Stahlberg and Byrne, 2019; Holtzman et al., 2020; Meister et al., 2020; Zhang et al., 2021).”
Yang, Y., Chern, E., Qiu, X., Neubig, G., & Liu, P. (2024) · neural information processing systems · doi:10.52202/079017-2030 · Cited by 201
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.620 · Cited by 200
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.15294 · Cited by 200
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., & Bing, L. (2023) · arXiv · doi:10.48550/arxiv.2311.16922 · Cited by 199
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., & Bing, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01316 · Cited by 199
FELM: Benchmarking Factuality Evaluation of Large Language Models
Chen, S., Zhao, Y., Zhang, J., Chern, I.-C., Gao, S., Liu, P., & He, J. (2023) · neural information processing systems · doi:10.52202/075280-1927 · Cited by 198
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197 · cited 1×
“…such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al., 2022), data-to-text(Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al., 2021).”
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Zhu, W., Liu, H., Dong, Q., Xu, J., Huang, S., Kong, L., Chen, J., & Li, L. (2023) · arXiv · doi:10.48550/arxiv.2304.04675 · Cited by 195
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.847 · Cited by 194
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · neural information processing systems · doi:10.52202/075280-3155 · Cited by 191
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024) · arXiv · doi:10.48550/arxiv.2401.06209 · Cited by 190
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00914 · Cited by 189
Knowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · arXiv · doi:10.48550/arxiv.2306.03341 · Cited by 184
Hallucination: Philosophy and Psychology
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9780262019200.001.0001 · Cited by 183
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P., & Sun, L. (2025) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 182
Instruction Tuning for Large Language Models: A Survey
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wang, G., et al. (2026) · ACM Computing Surveys · doi:10.1145/3777411 · Cited by 182
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Zhu, W., Liu, H., Dong, Q., Xu, J., Huang, S., Kong, L., Chen, J., & Li, L. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.176 · Cited by 181
Editing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180 · cited 2×
“Knowledge editing De Cao et al. (2021); Sinitsin et al. (2020) has garnered rising attention from researchers, which aims to rectify model behavior by incorporating additional knowledge.”
The Internal State of an LLM Knows When It's Lying
Azaria, A., & Mitchell, T. (2023) · arXiv · doi:10.48550/arxiv.2304.13734 · Cited by 179
QuestEval: Summarization Asks for Fact-based Evaluation
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.529 · Cited by 175 · cited 1×
“Notable implementations include (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021).”
Nationality Bias in Text Generation
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T.-H., & Wilson, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.9 · Cited by 175
The Internal State of an LLM Knows When It’s Lying
Azaria, A., & Mitchell, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.68 · Cited by 175 · cited 3×
“Several studies have demonstrated that LLM’s activations encapsulate an internal belief related to the truthfulness of its generated statements (Burns et al., 2022; Azaria and Mitchell, 2023).”
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.825 · Cited by 168
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 168 · cited 1×
“…the substantial advancements chain-of-thought prompting has brought to LLMs in multi-step reasoning Wei et al. (2022), numerous studies (Yao et al., 2022; Trivedi et al., 2023; He et al., 2023) Beyond multi-step reasoning tasks, Jiang et al. (2023) shift their emphasis to long-form generation.”
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., & Zhou, M. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06353 · Cited by 167
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01363 · Cited by 166
(2023) · doi:10.48550/arxiv.2310.14566 · Cited by 166
GPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.365 · Cited by 166
Theoretical Limitations of Self-Attention in Neural Sequence Models
Hahn, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00306 · Cited by 165 · cited 1×
“A potential cause is the limitations of soft attention (Hahn, 2020; Chi-ang and Cholak, 2022), where attention becomes diluted across positions as sequence length increases.”
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164
SCOTT: Self-Consistent Chain-of-Thought DistillationPreprint
Wang, P., Wang, Z., Li, Z., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.01879 · Cited by 164
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162 · cited 1×
“Thus, Goodrich et al. (2019) focus on the overlap of relation tuples and introduce a metric that computes the overlap of relation tuples extracted using trained end-to-end fact extraction models.”
Research works printed but not linked: 33
- ArXiv. 2023. arxiv dataset. Retrieved from https://www.kaggle.com/datasets/Cornell-University/arxiv/versions/134
- Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems. Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.), 1171–1179. Retrieved from https://proceedings.neurips.cc/paper/2015/hash/e995f98d56967d946471af29d7bf99f1-Abstract.html
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of theInternational Conference on Machine Learning (ICML ’22). Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, PMLR, 2206–2240. Retrieved from https://proceedings.mlr.press/v162/borgeaud22a.html
- 10.5555/3495724.3495883 · doi:10.5555/3495724.3495883
- 10.5555/3600270.3602050 · doi:10.5555/3600270.3602050
- Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems. Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.), 4299–4307. Retrieved from https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html
- Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In Proceedings of the 8th International Conference on Learning Representations (ICLR ’20). OpenReview.net. Retrieved from https://openreview.net/forum?id=H1edEyBKDS
- Bikash Gyawali, Lucas Anastasiou, and Petr Knoth. 2020. Deduplication of scholarly documents using locality sensitive hashing and word embeddings. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, 901–910. Retrieved from https://aclanthology.org/2020.lrec-1.113
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the 8th International Conference on Learning Representations (ICLR ’20). OpenReview.net. Retrieved from https://openreview.net/forum?id=rygGQyrFvH
- Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023d. Transformer-Patcher: One mistake worth one neuron. In Proceedings of the 11th International Conference on Learning Representations (ICLR ’23). OpenReview.net. Retrieved from https://openreview.net/pdf?id=4oYUGeGBPm
- Gautier Izacard Mathilde Caron Lucas Hosseini Sebastian Riedel Piotr Bojanowski Armand Joulin and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research 2022 (2022). Retrieved from https://openreview.net/forum?id=jKN1pXi7b0
- Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the International Conference on Machine Learning (ICML ’23). Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, PMLR, 15696–15707. Retrieved from https://proceedings.mlr.press/v202/kandpal23a.html
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In Proceedings of the 8th International Conference on Learning Representations (ICLR ’20). OpenReview.net. Retrieved from https://openreview.net/forum?id=HklBjCEKvH
- 10.5555/3600270.3601883 · doi:10.5555/3600270.3601883
- Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems. Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.), 6402–6413. Retrieved from https://proceedings.neurips.cc/paper/2017/hash/9ef2ed4b7fd2c810847ffa5fa85bce38-Abstract.html
- 10.5555/3600270.3602776 · doi:10.5555/3600270.3602776
- 10.5555/3495724.3496517 · doi:10.5555/3495724.3496517
- Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. 2023. GPT4Rec: A generative framework for personalized recommendation and user interests interpretation. In Proceedings of the SIGIR Workshop on eCommerce Co-located with the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 23). Surya Kallumadi, Yubin Kim, Tracy Holloway King, Shervin Malmasi, Maarten de Rijke, and Jacopo Tagliabue (Eds.), CEUR Workshop Proceedings, Vol. 3589, CEUR-WS.org. Retrieved from https://ceur-ws.org/Vol-3589/paper_2.pdf
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74–81. Retrieved from https://aclanthology.org/W04-1013
- 10.5555/3600270.3601532 · doi:10.5555/3600270.3601532
- Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In Proceedings of the 11th International Conference on Learning Representations (ICLR ’23). OpenReview.net. Retrieved from https://openreview.net/pdf?id=MkbcAHIYgyS
- Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. In Proceedings of the 10th International Conference on Learning Representations (ICLR ’22). OpenReview.net. Retrieved from https://openreview.net/forum?id=0DcZxeWfOPt
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. Retrieved from http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In Proceedings of the8th International Conference on Learning Representations (ICLR ’20). OpenReview.net. Retrieved from https://openreview.net/forum?id=HJedXaEtvS
- 10.5555/3495724.3495977 · doi:10.5555/3495724.3495977
- Chenmien Tan, Ge Zhang, and Jie Fu. 2024. Massive editing for large language model via meta learning. In Proceedings of the 12th International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=L6L1CJQ2PE
- Zirui Wang Jiahui Yu Adams Wei Yu Zihang Dai Yulia Tsvetkov and Yuan Cao. 2022. SimVLM: Simple visual language model pretraining with weak supervision. In Proceedings of the 10th International Conference on Learning Representations (ICLR ’22). OpenReview.net. Retrieved from https://openreview.net/forum?id=GUrhfTuf_3
- 10.5555/3600270.3602070 · doi:10.5555/3600270.3602070
- Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In Proceedings of the6th International Conference on Learning Representations (ICLR ’18). OpenReview.net. Retrieved from https://openreview.net/forum?id=HkwZSG-CZ
- Zhilin Yang, Thang Luong, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Mixtape: Breaking the softmax bottleneck efficiently. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS ’19). Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.), 15922–15930. Retrieved from https://proceedings.neurips.cc/paper/2019/hash/512fc3c5227f637e41437c999a2d3169-Abstract.html
- Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS ’21). Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.), 27263–27277. Retrieved from https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Abstract.html
- Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021. Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval). Association for Computational Linguistics, 25–33. Retrieved from https://aclanthology.org/2021.humeval-1.3
Cited by
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation
Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01670-7 · Cited by 423
Large Language Models for Mental Health Applications: Systematic Review
Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024) · JMIR Mental Health · doi:10.2196/57400 · Cited by 269
“Furthermore, hallucinations may stem from overfitting the training data or a lack of context understanding [134].”
BackgroundThe landscape of agentic reinforcement learning for LLMs: a survey
Zhang, G., Geng, H., Yu, X., Yin, Z., Zhang, Z., Tan, Z., Zhou, H., Li, Z., Xue, X., Li, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.02547 · Cited by 229
Self-citation
Metadata
- Authors
- Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & Liu, T.
- Affiliations
- Harbin Institute of Technology (CN)
- Huawei Technologies (China) (CN)
- ORCID
- Venue
- ACM Transactions on Information Systems
- Published
- 2025-01-24
- Volume
- 43
- Issue
- 2
- Pages
- 1-55
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W4404534210
- Semantic Scholar
- 1e909e2a8cdacdcdff125ebcc566f37cb869a1c8
- Cited by
- 4,688
- References
- 229
- Citations by year
- 2026: 2433 · 2025: 1643
- FWCI
- 504.87
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- LLM hallucination, large language models, hallucination taxonomy, hallucination detection, hallucination mitigation, retrieval-augmented generation, information retrieval systems, factual reliability, large vision-language models, knowledge boundaries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.