KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Open accessAbstract
The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For \textbf{ability modeling}, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. (2) For \textbf{data}, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For \textbf{evaluation criteria}, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate $28$ open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset and open-participation leaderboard are publicly released at https://kola.xlore.cn and will be continuously updated to provide references for developing LLMs and knowledge-related systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
According to KoLA benchmark, even top-ranked GPT4 achieves 0.012 Precision and 0.013 Recall on Named Entity Recognition, far short of task-specific models PL-Marker (0.712 Precision, 0.706 Recall).
“According to the KoLA [84], a benchmark for evaluating word knowledge of LLMs, even the top-ranked GPT4 just achieves 0.012 in Precision and 0.013 in Recall on the task Named Entity Recognition , which falls far short of the performance (0.712 in Precision and 0.706 in Recall) of the task-specific models PL-Marker [85].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [84]
Table 32 (absolute performance on FewNERD task 2-4; page unavailable) reports GPT-4 precision of 12.4% and recall of 14.5%, not the claimed 0.012 and 0.013. Our copy does not report the stated PL-Marker comparison. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“6 million English articles, which has been used in pre-training by numerous pre-trained models since BERT (Devlin et al., 2019; Brown et al., 2020; Shuster et al., 2022) and is widely included in open pre-training corpora (Gao et al., 2021).”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“…2023), ChatGLM2-32k (6B) (Zeng et al., 2022), Internlm-chat-8k (7B) (Team, 2023); (2) API service : GPT-3 curie v1 (6.7B) 6 and davinci v1 (175B) (Brown et al., 2020), InstructGPT curie v1 (6.7B*) 6 and davinci v2 (175B*) (Ouyang et al., 2022), ChatGLM (130B) (Zeng et al., 2022), Cohere-command…”
BackgroundA revision of bloom's taxonomy: an overview
Krathwohl, D. R. (2002) · Theory Into Practice · doi:10.1207/s15430421tip4104_2 · Cited by 8,132 · cited 3× · see citing works
“Knowledge Creating Tasks : As the highest level of Bloom’s Cognitive Taxonomy (Krathwohl, 2002), how to evaluate knowledge creation is a long-standing open and challenging question.”
BackgroundMethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 3× · see citing works
“Furthermore, evaluating knowledge creation is particularly challenging as it involves distinguishing the correctly created knowledge and knowledge hallucinations (Ji et al., 2023).”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2× · see citing works
“…GPT-JT (6B) (Computer, 2022), GPT-NeoX (20B) (Black et al., 2022), BLOOM (7B) (Scao et al., 2022), T0++ (11B) (Bang et al., 2023), LLaMa (65B) (Touvron et al., 2023), GLM (130B) (Zeng et al., 2022), UL2 (20B) (Tay et al., 2022), FLAN-T5 (11B) (Chung et al., 2022), FLAN-UL2 (20B) (Research,…”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1× · see citing works
“Considering the extensive and profound natural language understanding and generation abilities exhibited by LLMs (Bubeck et al., 2023), the conventional benchmarks (Wang et al., 2018; 2019) focusing on relatively narrow and superficial abilities are no longer as helpful for testing them.”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2× · see citing works
“34 ), potentially demonstrating the widely discussed “alignment tax” (Ouyang et al., 2022).”
BackgroundJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1× · see citing works
“…(6B) (Du et al., 2022), Dolly-v2 (12B) (Conover et al., 2023),RedPajama-Instruct (7B) (Computer, 2023), Tulu (7B) (Wang et al., 2023), Vicuna (13B) (Zheng et al., 2023), Llama2-chat (7B) (Touvron et al., 2023), ChatGLM2-32k (6B) (Zeng et al., 2022), Internlm-chat-8k (7B) (Team, 2023); (2) API…”
Self-citationConstructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning StepsPreprint
Ho, X., Nguyen, A.-K. D., Sugawara, S., & Aizawa, A. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2011.01060 · Cited by 1,880 · cited 4× · see citing works
“Specifically, given a task set D = { d i }”
BackgroundMethodInfluentialLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 3× · see citing works
“Hence various knowledge-intensive tasks (Petroni et al., 2019; 2021) are proposed to examine language models’ knowledge-related abilities.”
BackgroundInfluentialHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 5× · see citing works
“Consequently, this level is evaluated by various knowledge reasoning tasks (Yang et al., 2018; Trivedi et al., 2022; Cao et al., 2022).”
BackgroundMethodInfluentialSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2× · see citing works
“…and Out-of-Distribution evolving settings suggests a potential enhancement in the model’s generalization capability and may support the opinion about the model’s acquisition of divergent and reasoning abilities that go beyond simple data fitting (Bubeck et al., 2023; Zhong et al., 2023).”
BackgroundScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1× · see citing works
“…T0++ (11B) (Bang et al., 2023), LLaMa (65B) (Touvron et al., 2023), GLM (130B) (Zeng et al., 2022), UL2 (20B) (Tay et al., 2022), FLAN-T5 (11B) (Chung et al., 2022), FLAN-UL2 (20B) (Research, 2022), Alpaca (7B) (Taori et al., 2023), ChatGLM (6B) (Du et al., 2022), Dolly-v2 (12B) (Conover et…”
Self-citationA note on the evaluation of generative modelsPreprint
Theis, L., van den Oord, A., & Bethge, M. (2015) · International Conference on Learning Representations · doi:10.48550/arxiv.1511.01844 · Cited by 1,345 · cited 4× · see citing works
“We believe this metric can contribute to future explorations on the assessment of generation abilities (Theis et al., 2016; Pillutla et al., 2021; Fu et al., 2023).”
BackgroundInfluentialNormalization: A Preprocessing Stage
Patro, S., & Sahu, K. K. (2015) · arXiv.org · doi:10.48550/arxiv.1503.06462 · Cited by 1,341 · cited 1×
“Subsequently, we apply Min-Max scaling (Patro & Sahu, 2015) to adjust all the results to the range of [0 , 100] , further enhancing the correlation and readability of scores across tasks.”
MethodSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 2× · see citing works
“Considering the extensive and profound natural language understanding and generation abilities exhibited by LLMs (Bubeck et al., 2023), the conventional benchmarks (Wang et al., 2018; 2019) focusing on relatively narrow and superficial abilities are no longer as helpful for testing them.”
BackgroundMethodGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · cited 2× · see citing works
“…UL2 (20B) (Tay et al., 2022), FLAN-T5 (11B) (Chung et al., 2022), FLAN-UL2 (20B) (Research, 2022), Alpaca (7B) (Taori et al., 2023), ChatGLM (6B) (Du et al., 2022), Dolly-v2 (12B) (Conover et al., 2023),RedPajama-Instruct (7B) (Computer, 2023), Tulu (7B) (Wang et al., 2023), Vicuna (13B) (Zheng…”
Self-citationKILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., Cao, N. D., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2009.02252 · Cited by 940 · cited 1×
“Hence various knowledge-intensive tasks [20, 21] are proposed to examine language models’ knowledge-related abilities.”
BackgroundAssessing Generative Models via Precision and RecallPreprint
Sajjadi, M. S. M., Bachem, O., Lučić, M., Bousquet, O., & Gelly, S. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1806.00035 · Cited by 815 · cited 1× · see citing works
“Moreover, there are many well-known issues (Theis et al., 2016; Sajjadi et al., 2018; Ji et al., 2023) for evaluating tasks with large search spaces like the generative tasks.”
BackgroundCryptosporidium Parvum Genome Project
Abrahamsen, M. (2001) · Comparative and Functional Genomics · doi:10.1002/cfg.67 · Cited by 777
Defining higher order thinking
Lewis, A., & Smith, D. (1993) · Theory Into Practice · doi:10.1080/00405849309543588 · Cited by 772 · cited 1× · see citing works
“Such viewpoints have also been upheld by cognitive scientists for several decades, giving rise to a series of cognitive learning theories (Lewis & Smith, 1993).”
BackgroundReliable Fidelity and Diversity Metrics for Generative ModelsPreprint
Naeem, M. F., Oh, S. J., Choi, Y., Uh, Y., & Yoo, J. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2002.09797 · Cited by 705 · cited 1× · see citing works
“Traditional text generation evaluation metrics (Theis et al., 2016) are based on textual similarities between model-generated content and human-written references, which do not solely focus on knowledge creation ability but cover other skills, such as text style and fluency (Naeem et al., 2020).”
BackgroundPurposely Teaching for the Promotion of Higher-order Thinking Skills: A Case of Critical Thinking
Barak, M., Ben‐Chaim, D., & Uri, Z. (2007) · Research in Science Education · doi:10.1007/s11165-006-9029-2 · Cited by 697 · cited 1× · see citing works
“Considering the ongoing debates surrounding high-order thinking (Miri et al., 2007; Collins, 2014), we simplify and select four widely accepted processes in Bloom’s taxonomy (Krathwohl, 2002) for organizing the tasks in KoLA benchmark.”
MethodBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · cited 4× · see citing works
“…(6B) (Wang & Komatsuzaki, 2021), GPT-JT (6B) (Computer, 2022), GPT-NeoX (20B) (Black et al., 2022), BLOOM (7B) (Scao et al., 2022), T0++ (11B) (Bang et al., 2023), LLaMa (65B) (Touvron et al., 2023), GLM (130B) (Zeng et al., 2022), UL2 (20B) (Tay et al., 2022), FLAN-T5 (11B) (Chung et al.,…”
MethodInfluentialKEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00360 · Cited by 676 · cited 5× · see citing works
“Given a triplet in Wikidata5M (Wang et al., 2021a), we transform it into a sentence with a relation-specific template and let LLMs complete its tail entity.”
MethodInfluentialSelf-citationA Knowledge-Grounded Neural Conversation Model
Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, W., Gao, J., Yih, W.-T., & Galley, M. (2017) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1702.01932 · Cited by 605 · cited 1×
“The ∂ ( T k , R ) takes inspiration from the knowledge-grounded generation tasks (Chen et al., 2020; Ghazvininejad et al., 2018).”
BackgroundBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 3× · see citing works
“To this end, various benchmarks are proposed, focusing on extending the evaluation scope to cover broader abilities (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023) or more challenging tasks (Srivastava et al., 2022; Suzgun et al., 2022).”
BackgroundSelf-citationDocRED: A Large-Scale Document-Level Relation Extraction Dataset
Yao, Y., Ye, D., Li, P., Han, X., Lin, Y., Liu, Z., Liu, Z., Huang, L., Zhou, J., & Sun, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1074 · Cited by 550 · cited 2× · see citing works
“Yuan Yao provided the test set of DocRED (Yao et al., 2019) and Ning Ding helped the construction of FewNERD (Ding et al., 2021) test set.”
BackgroundMethodSelf-citationChameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsPreprint
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y., Zhu, S.-C., & Gao, J.-F. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2304.09842 · Cited by 534 · cited 1× · see citing works
“This differs from several recent studies (Lu et al., 2023; Mialon et al., 2023), which cover more general reasoning, such as mathematical reasoning.”
ResultHow Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.04751 · Cited by 532 · cited 1×
“…(Taori et al., 2023), ChatGLM (6B) (Du et al., 2022), Dolly-v2 (12B) (Conover et al., 2023),RedPajama-Instruct (7B) (Computer, 2023), Tulu (7B) (Wang et al., 2023), Vicuna (13B) (Zheng et al., 2023), Llama2-chat (7B) (Touvron et al., 2023), ChatGLM2-32k (6B) (Zeng et al., 2022),…”
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 1× · see citing works
“6 million English articles, which has been used in pre-training by numerous pre-trained models since BERT (Devlin et al., 2019; Brown et al., 2020; Shuster et al., 2022) and is widely included in open pre-training corpora (Gao et al., 2021).”
BackgroundHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 1× · see citing works
“This suggests that model size has an obvious positive impact on memorizing seen knowledge, which corroborates some of the viewpoints from previous studies (Liang et al., 2022).”
ResultA Knowledge-Grounded Neural Conversation Model
Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J., Yih, W.-T., & Galley, M. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11977 · Cited by 457 · see citing works
GPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.365 · Cited by 446 · cited 2× · see citing works
“We believe this metric can contribute to future explorations on the assessment of generation abilities [10, 63, 64].”
BackgroundGPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2302.04166 · Cited by 446 · see citing works
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling
Zhou, W., Huang, K., Ma, T., & Huang, J. (2020) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2010.11304 · Cited by 407 · cited 2×
“We also provide a data access API, which assists authorized users in getting the evolving data and results of previous seasons.”
BackgroundMethodGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q. G., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 407 · see citing works
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.200 · Cited by 357 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling
Zhou, W., Huang, K., Ma, T., & Huang, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17717 · Cited by 334 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 7× · see citing works
“A.1 L IMITATION The major limitation of KoLA is that our coverage is not as extensive as some other recent works (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023).”
BackgroundMethodResultInfluentialConstructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287 · see citing works
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · cited 2× · see citing works
“Furthermore, the (3-3) MuSiQue (Trivedi et al., 2022) dataset tackles the challenges of shortcuts and naturalness simultaneously.”
BackgroundGLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281 · cited 4× · see citing works
“…(7B) (Computer, 2023), Tulu (7B) (Wang et al., 2023), Vicuna (13B) (Zheng et al., 2023), Llama2-chat (7B) (Touvron et al., 2023), ChatGLM2-32k (6B) (Zeng et al., 2022), Internlm-chat-8k (7B) (Team, 2023); (2) API service : GPT-3 curie v1 (6.7B) 6 and davinci v1 (175B) (Brown et al., 2020),…”
BackgroundInfluentialSelf-citationHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Few-NERD: A Few-shot Named Entity Recognition Dataset
Ding, N., Xu, G., Chen, Y., Wang, X., Han, X., Xie, P., Zheng, H.-T., & Liu, Z. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.248 · Cited by 245 · cited 3× · see citing works
“Yuan Yao provided the test set of DocRED (Yao et al., 2019) and Ning Ding helped the construction of FewNERD (Ding et al., 2021) test set.”
BackgroundMethodSelf-citationMAVEN: A Massive General Domain Event Detection DatasetPreprint
Wang, X., Wang, Z., Han, X., Jiang, W., Han, R., Liu, Z., Li, J.-Z., Li, P., Lin, Y., & Zhou, J. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2004.13590 · Cited by 240 · cited 7× · see citing works
“For Task (2), we specifically select 100 articles from the MAVEN and Evolving datasets, ensuring they possess extensive knowledge (i.e., a substantial number of entities and event triggers that are not within the lowest 20 % frequency range).”
BackgroundMethodInfluentialSelf-citationAGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 8× · see citing works
“A.1 L IMITATION The major limitation of KoLA is that our coverage is not as extensive as some other recent works (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023).”
BackgroundResultInfluentialDyck, P., Boes, C., Mulder, D. M., Millikan, C., Windebank, A., Dyck, P., & Espinosa, R. (2005) · Journal of the peripheral nervous system · doi:10.1111/j.1085-9489.2005.0010206.x · Cited by 207 · cited 1× · see citing works
“Therefore, we propose to introduce standardized scores (Dyck et al., 2005) to enhance the applicability of KoLA results.”
MethodUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 194 · see citing works
Self-citationRetrieval Augmentation Reduces Hallucination in ConversationPreprint
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07567 · Cited by 172 · cited 1× · see citing works
“Moreover, there are many well-known issues [10, 11, 12, 13] awaiting resolutions for evaluating tasks with large search spaces like the generative tasks.”
BackgroundTime-Aware Language Models as Temporal Knowledge Bases
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00459 · Cited by 169 · cited 1× · see citing works
“We believe the ideal approach is to do evaluations on newly emerging data and maintain a continuously evolving benchmark, like the attempts that include time-sensitive evolving data (Kasai et al., 2022; Dhingra et al., 2022).”
BackgroundMAVEN: A Massive General Domain Event Detection Dataset
Wang, X., Wang, Z., Han, X., Jiang, W., Han, R., Liu, Z., Li, J., Li, P., Lin, Y., & Zhou, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.129 · Cited by 166 · see citing works
Self-citationToward Gender-Inclusive Coreference Resolution
Cao, Y. T., & Daumé, H. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1910.13913 · Cited by 162 · cited 1×
“Although AO3 data has been used in some previous works Cao & Daumé (2020); Yoder et al. (2021); Sun et al. (2022); Krishna et al. (2022), there remains some ambiguity regarding its copyright status.”
MethodAugmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160 · cited 1× · see citing works
“This differs from several recent studies (Lu et al., 2023; Mialon et al., 2023), which cover more general reasoning, such as mathematical reasoning.”
ResultMAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersPreprint
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., & Harchaoui, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01454 · Cited by 128 · cited 1× · see citing works
“We believe this metric can contribute to future explorations on the assessment of generation abilities (Theis et al., 2016; Pillutla et al., 2021; Fu et al., 2023).”
BackgroundCLEVE: Contrastive Pre-training for Event Extraction
Wang, Z., Wang, X., Han, X., Lin, Y., Hou, L., Liu, Z., Li, P., Li, J., & Zhou, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.491 · Cited by 121 · cited 1× · see citing works
“Specifically, we replace the backbone model with CLEVE (Wang et al., 2021b) for event detection, which achieves a Recall of 81 .”
MethodSelf-citationWang, X., Chen, Y., Ding, N., Peng, H., Wang, Z., Lin, Y.-K., Han, X., Hou, L., Li, J.-Z., Liu, Z.-Y., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2211.07342 · Cited by 120 · cited 1×
“Extraction involves the undisclosed test set from MAVEN-ERE [40], which consists of 113k examples of coreference, temporal, causal, and subevent relations between events.”
MethodSelf-citationKGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation
Chen, W., Su, Y., Yan, X., & Wang, W. Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.697 · Cited by 120 · see citing works
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.03188 · Cited by 95 · cited 1× · see citing works
“6 million English articles, which has been used in pre-training by numerous pre-trained models since BERT (Devlin et al., 2019; Brown et al., 2020; Shuster et al., 2022) and is widely included in open pre-training corpora (Gao et al., 2021).”
BackgroundRankGen: Improving Text Generation with Large Ranking ModelsPreprint
Krishna, K., Chang, Y., Wieting, J., & Iyyer, M. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.09726 · Cited by 94 · cited 1× · see citing works
“Hanming Li deployed and conduct experiments of GPT-J (6B), GPT-JT (6B), GPT-NeoX (20B), BLOOM (7B), T0++ (11B), LLaMa (65B), GLM (130B), UL2 (20B), FLAN-T5 (11B), Alpaca (7B), FLAN-UL2 (20B), ChatGLM (6B).”
Cao, S., Shi, J., Pan, L., Nie, L., Xiang, Y., Hou, L., Li, J., He, B., & Zhang, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.422 · Cited by 85 · cited 2× · see citing works
“Consequently, this level is evaluated by various knowledge reasoning tasks (Yang et al., 2018; Trivedi et al., 2022; Cao et al., 2022).”
BackgroundSelf-citationGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · cited 1× · see citing works
“…LLMs of two categories: (1) Open-source Model , including GPT-J (6B) (Wang & Komatsuzaki, 2021), GPT-JT (6B) (Computer, 2022), GPT-NeoX (20B) (Black et al., 2022), BLOOM (7B) (Scao et al., 2022), T0++ (11B) (Bang et al., 2023), LLaMa (65B) (Touvron et al., 2023), GLM (130B) (Zeng et al.,…”
Belz, A., Thomson, C., Reiter, E., Abercrombie, G., Alonso-Moral, J., Arvan, M., Braggaar, A., Cieliebak, M., Clark, E., van Deemter, K., et al. (2023) · First Workshop on Insights from Negative Results in NLP · doi:10.48550/arxiv.2305.01633 · Cited by 53 · cited 1× · see citing works
“Evaluations for related abilities still heavily rely on human evaluation, which is time-consuming and not easily reproducible [14, 15].”
BackgroundRealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N. A., Choi, Y., & Inui, K. (2022) · arXiv · doi:10.48550/arxiv.2207.13332 · Cited by 52 · cited 1× · see citing works
“We believe the ideal approach is to do evaluations on newly emerging data and maintain a continuously evolving benchmark, like the attempts that include time-sensitive evolving data (Kasai et al., 2022; Dhingra et al., 2022).”
BackgroundRevisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE?
Akter, M., Bansal, N., & Karmaker, S. (2022) · Findings · doi:10.18653/v1/2022.findings-acl.122 · Cited by 52 · cited 1× · see citing works
“This approach offers an effective assessment of knowledge creation abilities compared to traditional text generation metrics encompassing many other factors (Akter et al., 2022).”
BackgroundChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 4× · see citing works
“To this end, various benchmarks are proposed, focusing on extending the evaluation scope to cover broader abilities (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023) or more challenging tasks (Srivastava et al., 2022; Suzgun et al., 2022).”
BackgroundInfluentialWang, X., Chen, Y., Ding, N., Peng, H. M., Wang, Z., Lin, Y., Han, X., Hou, L., Li, J., Liu, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.60 · Cited by 43 · see citing works
Self-citationXLORE2: Large-scale Cross-lingual Knowledge Graph Construction and Application
Jin, H., Li, C., Zhang, J., Hou, L., Li, J.-Z., & Zhang, P. (2019) · Data Intelligence · doi:10.1162/dint_a_00003 · Cited by 40 · cited 14× · see citing works
“Considering that many LLMs state they can only provide answers based on “Content before 2021” 3 , we select Wikidata5M (Wang et al., 2021a), a high-quality subset of Wikidata, as the basis, which allows linking to the 2019 version of Wikipedia dump, thus enabling the selection or reconstruction of downstream tasks’ da…”
MethodInfluentialSelf-citationCOPEN: Probing Conceptual Knowledge in Pre-trained Language Models
Peng, H., Wang, X., Hu, S.-D., Jin, H., Hou, L., Li, J.-Z., Liu, Z.-Y., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2211.04079 · Cited by 38 · cited 7×
“Table 10: The instruction and an example of Task 2-3 COPEN-CiC, KU.”
BackgroundInfluentialSelf-citationQuantified Reproducibility Assessment of NLP Results
Belz, A., Popovi'c, M., & Mille, S. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2204.05961 · Cited by 35 · cited 1×
“Evaluations for related abilities still heavily rely on human evaluation, which is time-consuming and not easily reproducible (Belz et al., 2022; 2023).”
BackgroundYu, J.-F., Zhang, X., Xu, Y.-F., Lei, X., Guan, X., Zhang, J., Hou, L., Li, J.-Z., & Tang, J. (2022) · Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539135 · Cited by 23 · cited 2× · see citing works
“We present a feasible proposal based on knowledge-grounded text generation tasks [32, 45].”
MethodSelf-citationChapterBreak: A Challenge Dataset for Long-Range Language ModelsPreprint
Sun, S., Thai, K., & Iyyer, M. (2022) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2204.10878 · Cited by 22 · cited 1× · see citing works
“Hanming Li deployed and conduct experiments of GPT-J (6B), GPT-JT (6B), GPT-NeoX (20B), BLOOM (7B), T0++ (11B), LLaMa (65B), GLM (130B), UL2 (20B), FLAN-T5 (11B), Alpaca (7B), FLAN-UL2 (20B), ChatGLM (6B).”
Toward Gender-Inclusive Coreference Resolution
Cao, Y. T., & Daumé, H., III (2020) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2020.acl-main.418 · Cited by 22 · see citing works
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
Peng, H. M., Wang, X., Hu, S., Jin, H., Hou, L., Li, J., Liu, Z., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.335 · Cited by 22 · see citing works
Self-citationQuantified Reproducibility Assessment of NLP Results
Belz, A., Popovic, M., & Mille, S. (2022) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2022.acl-long.2 · Cited by 20 · see citing works
Belz, A., Thomson, C., & Reiter, E. (2023) · The Fourth Workshop on Insights from Negative Results in NLP · doi:10.18653/v1/2023.insights-1.1 · Cited by 18 · see citing works
FanfictionNLP: A Text Processing Pipeline for Fanfiction
Yoder, M. M., Khosla, S., Shen, Q., Naik, A., Jin, H., Muralidharan, H., & Rosé, C. P. (2021) · NUSE · doi:10.18653/v1/2021.nuse-1.2 · Cited by 14 · cited 1× · see citing works
“Although AO3 data has been used in some previous works Cao & Daumé (2020); Yoder et al. (2021); Sun et al. (2022); Krishna et al. (2022), there remains some ambiguity regarding its copyright status.”
MethodSelf-citationKoRC: Knowledge oriented Reading Comprehension Benchmark for Deep Text Understanding
Yao, Z., Liu, Y.-T., Lv, X., Cao, S., Yu, J.-F., Hou, L., & Li, J.-Z. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2307.03115 · Cited by 11 · cited 3×
“A NSWER : before Table 20: The instruction and an example of Task 3-5 KoRC, KA.”
BackgroundInfluentialSelf-citationChapterBreak: A Challenge Dataset for Long-Range Language Models
Sun, S., Thai, K., & Iyyer, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.271 · Cited by 7 · see citing works
KoRC: Knowledge Oriented Reading Comprehension Benchmark for Deep Text Understanding
Yao, Z., Liu, Y., Lv, X., Cao, S., Yu, J., Li, J., & Hou, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.743 · Cited by 2 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 6×
“Recent remarkable breakthroughs achieved by large language models (LLMs) like GPT-4 [1] have elicited widespread astonishment.”
BackgroundMethodInfluentialSelf-citationBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100 · cited 1×
“…Open-source Model , including GPT-J (6B) (Wang & Komatsuzaki, 2021), GPT-JT (6B) (Computer, 2022), GPT-NeoX (20B) (Black et al., 2022), BLOOM (7B) (Scao et al., 2022), T0++ (11B) (Bang et al., 2023), LLaMa (65B) (Touvron et al., 2023), GLM (130B) (Zeng et al., 2022), UL2 (20B) (Tay et al., 2022),…”
Self-citationC-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08322 · cited 7×
“A.1 L IMITATION The major limitation of KoLA is that our coverage is not as extensive as some other recent works (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023).”
BackgroundResultInfluentialSelf-citationXLORE2: Large-Scale Cross-Lingual Knowledge Graph Construction and Application
Jin, H. J. H., Li, C. L. C., Zhang, J. Z. J., Hou, L. H. L., Li, J. L. J., & Zhang, P. Z. P. (n.d.) · Science Data Bank Datasets · doi:10.11922/sciencedb.j00104.00022
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Robyn Collins. Skills for the 21st century: teaching higher-order thinking. Curriculum & Leadership Journal, 12(14), 2014.
- Edward A Feigenbaum. The art of artificial intelligence: Themes and case studies of knowledge engineering. In Proceedings of IJCAI, volume 2. Boston, 1977.
- Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, 2004. URL https://aclanthology.org/W04-1013.
- Mary Lou Maher. Evaluating creativity in humans, computers, and collectively intelligent systems. In Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design, pp. 22-28, 2010.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationSiren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534 · see citing works
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
“According to the KoLA [67], a benchmark for evaluating word knowledge of LLMs, even the top-ranked”
InfluentialSelf-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Can LLM-Generated Misinformation Be Detected?Preprint
Chen, C., & Shu, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.13788 · Cited by 135 · see citing works
WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation
Niu, Y.-W., Ning, M.-N., Zheng, M., Lin, B., Jin, P., Liao, J., Feng, C., Ning, K., Zhu, B., & Yuan, L. (2025) · arXiv.org · doi:10.48550/arxiv.2503.07265 · Cited by 230
“This shortcoming mainly stems from their limited ability to model the information behind world knowl-* Equal contribution † Corresponding Author edge [57] - this information constitutes the massive facts and complex relationships required for real-world understanding.”
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
LawBench: Benchmarking Legal Knowledge of Large Language ModelsPreprint
Fei, Z.-W., Shen, X., Zhu, D., Zhou, F., Han, Z., Zhang, S., Chen, K., Shen, Z.-W., & Ge, J.-D. (2023) · arXiv.org · doi:10.48550/arxiv.2309.16289 · Cited by 176 · see citing works
“…7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios to the extent possible, KOLA [73] which focuses on knowledge-oriented assessment under four-level taxonomy of knowledge-related abilities, and MMBench [39] which is designed…”
BackgroundMethodInfluentialSurvey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174 · see citing works
“In addition, leveraging the vast amounts of training data, pretrained LLMs are equipped with a broad spectrum of world knowledge [39].”
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Chen, L., Wang, X., Huang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.09150
“1 shows that existing benchmarks are either close-ended (Huang et al. 2023; Zhong et al. 2023; Yu et al. 2023) or contain common and simple instructions (Srivastava et al.”
BackgroundSelf-citationThe benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110 · see citing works
Extrapolating Large Language Models to Non-English by Aligning LanguagesPreprint
Zhu, W., Lv, Y., Dong, Q., Yuan, F., Xu, J.-J., Huang, S., Kong, L., Chen, J.-J., & Li, L. (2023) · arXiv.org · doi:10.48550/arxiv.2308.04948 · Cited by 107 · see citing works
“With large-scale parameters (usually billions of neurons) being trained on a large-scale corpus (usually trillions of tokens), LLM can acquire extensive knowledge (Zhang et al., 2022; Yu et al., 2023).”
BackgroundSelf-citationRWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
Jin, Z.-R., Cao, P., Wang, C., He, Z., Yuan, H.-B., Li, J., Chen, Y., Liu, K., & Zhao, J. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.10890 · Cited by 105
“We conduct the forget assessment to evaluate the unlearning efficacy at both the knowledge memorization [67; 61] and knowledge manipulation abilities [3].”
MethodSelf-citationInadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“• KoLA [11]: The study acknowledged the inadequacy by incorporating evolving data sources, including non-English content like news articles and novels, to evaluate models’ understanding and generation abilities in diverse linguistic contexts.”
BackgroundMethodInfluentialDou, S.-H., Zhou, E.-Y., Liu, Y., Gao, S., Zhao, J., Shen, W., Zhou, Y., Xi, Z., Wang, X., Fan, X.-R., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.09979 · Cited by 96 · see citing works
Self-citationEnhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
Maniparambil, M., Vorster, C., Molloy, D., Murphy, N. A., McGuinness, K., & O'Connor, N. E. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00034 · Cited by 53 · see citing works
“We find that the performance improves with larger models which are more capable of memorizing accurate class information with less hallucination [33].”
BackgroundMethodInfluentialKnowledge Mechanisms in Large Language Models: A Survey and Perspective
Wang, M.-R., Yao, Y., Xu, Z., Qiao, S.-F., Deng, S.-M., Wang, P., Chen, X., Gu, J.-C., Jiang, Y., Xie, P.-J., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2407.15017 · Cited by 81
“…commonsense and concepts (Allen-Zhu Figure 3: The mechanism analysis for knowledge utilization across three levels: memorization, comprehension and application, and creation. and Li, 2023a; Yu et al., 2023a; Mahowald et al., 2023; Zhu and Li, 2023; Allen-Zhu and Li, 2023b, 2024; Cao et al., 2024a).”
Self-citationLLaRA: Supercharging Robot Learning Data for Vision-Language PolicyPreprint
Li, X., Mata, C., Park, J. S., Kahatapitiya, K., Jang, Y. S., Shang, J., Ranasinghe, K., Burgert, R., Cai, M., Lee, Y. J., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2406.20095 · Cited by 71 · see citing works
“The strong performance of such approaches can be attributed to the extensive world knowledge ( e.g ., understanding of physics, human common-sense) (Yu et al., 2023; Zhao et al., 2024) and powerful reasoning abilities (Creswell & Shanahan, 2022; Liu et al., 2023b) of underlying language models.”
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61 · see citing works
On the Challenges and Opportunities in Generative AI
Manduchi, L., Meister, C., Pandey, K., Bamler, R., Cotterell, R., Däubener, S., Fellenz, S., Fischer, A., Gärtner, T., Kirchler, M., et al. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2403.00025 · Cited by 60
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 58 · see citing works
“Xiang et al.[280] deploys an embodied agent in a world model, the simulator of VirtualHome [198], where the corresponding KoLA [305] 2024 ICLR Language Benchmark EWOK [112] 2024 arxiv Language Benchmark Geometry of Concepts [151] 2024 arxiv Language Analysis BLEnD [181] 2024 NeurIPS Language…”
PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain
Zhu, W., Wang, X., Zheng, H., Chen, M., & Tang, B. (2023) · arXiv.org · doi:10.48550/arxiv.2310.14151 · Cited by 55
“Probing the medical knowledge of LLMs Knowledge, the information encompassing facts, events, and skills, has long served as an central indicator for intelligence level of artificial intelligence (AI) (Yu et al., 2023).”
BackgroundSelf-citationResolving Knowledge Conflicts in Large Language ModelsPreprint
Wang, Y., Feng, S., Wang, H., Shi, W.-J., Balachandran, V., He, T., & Tsvetkov, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2310.00935 · Cited by 55 · see citing works
“Understanding and expanding the knowledge abilities of LLMs Previous works have demonstrated that LLMs have incorporated factual knowledge within their parameters and exhibit considerable potential in recalling factual information (Peters et al., 2018; Petroni et al., 2019; Yu et al., 2023; Mruthyunjaya et al., 2023).”
BackgroundLi, J., Shen, Y., Zhou, W., & Hui, C. H. E. N. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103665 · Cited by 26 · see citing works
“To evaluate SearchExpert’s reasoning-intensive multimedia search capabilities, we developed SearchExpertBench-25, a benchmark comprising 200 complex multiple-choice questions spanning financial and international news domains [23–25].”
Self-citationYin, X.-J., Zhang, X., Ruan, J., & Wan, X.-J. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.11493 · Cited by 49
Self-citationLiang, X., Song, S., Niu, S., Li, Z., Xiong, F., Tang, B., Wang, Y., He, D., Cheng, P., Wang, Z., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.288 · Cited by 32 · see citing works
“There are also works using automatic machine labeling [10], [35], [24], [6], [36].”
BackgroundMethodBenchmarking Agentic Workflow Generation
Qiao, S.-F., Fang, R.-N., Qiu, Z.-S., Wang, X.-B., Zhang, N.-Y., Jiang, Y., Xie, P.-J., Huang, F., & Chen, H.-J. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.07869 · Cited by 47
Self-citationToward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
Cao, Y.-X., Hong, S.-B., Li, X., Ying, J.-H., Ma, Y., Liang, H.-Y., Liu, Y.-T., Yao, Z., Wang, X.-Z., Huang, D., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.18838 · Cited by 42
Self-citationSECURE: Benchmarking Large Language Models for Cybersecurity
Bhusal, D., Alam, T., Nguyen, L., Mahara, A., Lightcap, Z., Frazier, R., Fieblinger, R., Torales, G. L., Blakely, B., & Rastogi, N. (2024) · Annual Computer Security Applications Conference · doi:10.1109/acsac63791.2024.00019 · Cited by 32 · see citing works
“Evaluation Benchmark: While there are various benchmarks like GLUE [6], MMLU [7], Helm [8] and KOLA [9] for evaluating general-purpose LLMs, comprehensive cybersecurity-specific benchmarks remain limited.”
InfluentialChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?Preprint
Chen, H., Jiao, F., Li, X., Qin, C., Ravaut, M., Zhao, R., Xiong, C., & Joty, S. R. (2023) · arXiv.org · doi:10.48550/arxiv.2311.16989 · Cited by 36 · see citing works
“• KoLA (Yu et al., 2023a) constructs a Knowledge-oriented LLM Assessment benchmark (KoLA) with three crucial factors: mimicking human cognition for ability modeling, using Wikipedia for data collection, and designing contrastive metrics for automatic hallucination evaluation.”
BackgroundA LLM-informed multi-agent AI system for drone-based visual inspection for infrastructure
Liu, J., Li, H., Chai, C., Chen, K., & Wang, D. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103643 · Cited by 32 · see citing works
Self-citationSupervised Knowledge Makes Large Language Models Better In-context LearnersPreprint
Yang, L., Zhang, S., Yu, Z., Bao, G., Wang, Y., Wang, J., Xu, R., Ye, W., Xie, X., Chen, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.15918 · Cited by 7 · see citing works
Quantifying uncert-AI-nty: Testing the accuracy of LLMs’ confidence judgments
Cash, T. N., Oppenheimer, D. M., Christie, S., & Devgan, M. (2025) · Memory & Cognition · doi:10.3758/s13421-025-01755-4 · Cited by 14 · see citing works
“This can be achieved through a variety of advanced statistical approaches (X. Jiang et al., 2012; Kuhn et al., 2023; Kumar et al., 2019; Liang et al., 2022; Wang, 2023; Xiao et al., 2022).”
BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language ModelsPreprint
Leong, W. Q., Ngui, J. G., Susanto, Y., Rengarajan, H., Sarveswaran, K., & Tjhi, W.-C. (2023) · arXiv.org · doi:10.48550/arxiv.2309.06085 · Cited by 23 · see citing works
“Unlike HELM [8], KoLA [51], BIG-Bench [7] and some other benchmarks, which explicitly test for the knowledge of language models, we were unable to include such tests in BHASA as datasets are scarce.”
Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving TestingPreprint
Jiang, H., Yi, X.-Y., Wei, Z.-H., Xiao, Z., Wang, S., & Xie, X. (2024) · arXiv.org · doi:10.48550/arxiv.2406.14230 · Cited by 22 · see citing works
Insights into Classifying and Mitigating LLMs' Hallucinations
Bruno, A., Mazzeo, P. L., Chetouani, A., Tliba, M., & Kerkouri, M. A. (2023) · AIxPAC@AI*IA · doi:10.48550/arxiv.2311.08117 · Cited by 22
“Yu et al. [18] tackled the mentioned [41] highlight the potential LLMs capabilities of constructing and inferring from Knowledge Graphs.”
BackgroundRe-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives
Wang, Z.-H., Zhao, S., Wang, Y., Huang, H., Shi, J.-X., Xie, S.-T., Wang, Z., Zhang, Y., Li, H., & Yan, J. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2408.06904 · Cited by 21
“To effectively deploy LLMs for domain-specific tasks, practitioners commonly utilize three primary methodologies: 1) Continuous pretraining [17, 21], which involves further training of pre-trained models on domain-specific data to refine their understanding of the domain’s unique characteristics.”
MethodSelf-citationMobileAIBench: Benchmarking LLMs and LMMs for On-Device Use CasesPreprint
Murthy, R., Yang, L., Tan, J.-T., Awalgaonkar, T., Zhou, Y., Heinecke, S., Desai, S., Wu, J., Xu, R., Tan, S., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.10290 · Cited by 21 · see citing works
“KoLA [45] uses Wikipedia and the continuously collected emerging corpora data to provides knowledge-oriented LLM assessment tasks.”
MethodSelf-citationPertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations
Li, J.-T., Hu, R.-J., Huang, K., Zhuang, Y., Liu, Q., Zhu, M., Shi, X., & Lin, W. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.19740 · Cited by 21
“Large language models (LLMs) are developing rapidly, and have shown excellent basic capabilities, such as reasoning [1, 2], planning [3] and world knowledge [4], in real-world tasks.”
Self-citationLLMs as Repositories of Factual Knowledge: Limitations and Solutions
Mousavi, S. M., Alghisi, S., & Riccardi, G. (2025) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslpro.2026.3680709 · Cited by 19 · see citing works
“An evaluation framework assessing models on memorization, understanding, application, and creation of knowledge was proposed in [28].”
Yu, X., Fu, J.-Q., Deng, R., & Han, W. (2024) · arXiv.org · doi:10.48550/arxiv.2403.19267 · Cited by 19 · see citing works
“In particular, LLMs have demonstrated effective performance for task-planning [18, 69, 38, 36, 17], and they possess substantial world knowledge [63].”
BackgroundOh, J., Kim, S., Seo, J., Wang, J.-D., Xu, R., Xie, X., & Whang, S. E. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.05266 · Cited by 18
“• Music [27]: We use a relation with the attributes music title , artist name , released year , and genre .”
Wen, J., Li, Z., Xi, M., & He, J. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3540820 · Cited by 20 · see citing works
“In addition, through training on massive datasets, pretrained LLMS have general world knowledge [34].”
Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies
Yu, J.-J., Wang, Y., & Ma, W. (2024) · Transportation Research Part E: Logistics and Transportation Review · doi:10.48550/arxiv.2410.10212 · Cited by 16
“Leveraging pre-trained world knowledge, LLMs exhibit notable in-context learning and reasoning abilities, surpassing previous AI technologies (Yu et al., 2023c; Huang and Chang, 2022; Wei et al., 2023).”
Self-citationDICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning
Tu, S., Zhu, K.-J., Bai, Y., Yao, Z.-J., Hou, L., & Li, J.-Z. (2024) · arXiv.org · doi:10.48550/arxiv.2406.04197 · Cited by 13
“For instance, the field primarily uses GSM8k [2] and MATH [3] to evaluate LLMs’ math reasoning abilities.”
MethodSelf-citationLangBridge: Interpreting Image as a Combination of Language Embeddings
Liao, J., Niu, Y.-W., Meng, F., Li, H., Tian, C.-Y., Du, Y., Xiong, Y., Li, D., Zhu, X.-Z., Yuan, L., et al. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.02205 · Cited by 12
“To overcome this limitation, visual encoders [27–29, 34, 47] have been integrated, serving as the “eyes" of large vision-language models to leverage LLMs’ extensive knowledge [26, 38, 45] and strong instruction-following capabilities.”
Self-citationNewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates
Deng, H., Jiao, W., Liu, X.-B., Zhang, M., & Tu, Z. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2410.20814 · Cited by 11
Liu, J.-X., Cao, S., Shi, J., Zhang, T.-J., Nie, L.-Y., Hu, L.-M., Hou, L., & Li, J.-Z. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2024.findings-acl.45 · Cited by 11
“Previous works have validated the ability of LLMs to memo-rize, understand and apply knowledge for reasoning (Yu et al., 2023b), and serve as agents to solve KBQA problem (Liu et al., 2023).”
BackgroundSelf-citationClimate Change from Large Language Models
Zhu, H., & Tiwari, P. (2023) · arXiv.org · doi:10.48550/arxiv.2312.11985 · Cited by 11
“KoLA [11] is a meticulously crafted knowledge-centric evaluation benchmark designed to assess the capabilities of LLMs.”
BackgroundDeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking
Wang, F., Yan, T.-W., Yang, Z., Hu, M.-H., Zhang, J., Luo, Z.-C., & Bai, X.-Y. (2025) · Information Processing & Management · doi:10.48550/arxiv.2508.15876 · Cited by 10
“In particular, LLMs have proven highly effective in task planning Gong et al. (2024), encode substantial world knowledge Yu et al. (2024), and exhibit impressive logical reasoning capabilities Xiong et al. (2025).”
Towards a Holistic Evaluation of LLMs on Factual Knowledge RecallPreprint
Yuan, J.-Q., Pan, L., Hang, C.-W., Guo, J., Jiang, J., Min, B.-N., Ng, P., & Wang, Z.-G. (2024) · arXiv.org · doi:10.48550/arxiv.2404.16164 · Cited by 10 · see citing works
Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks
Pink, M., Vo, V. A., Wu, Q., Mu, J.-N., Turek, J., Hasson, U., Norman, K. A., Michelmann, S., Huth, A., & Toneva, M. (2024) · arXiv.org · doi:10.48550/arxiv.2410.08133 · Cited by 9
Structure-aware Domain Knowledge Injection for Large Language Models
Liu, K., Chen, Z., Fu, Z., Zhang, W., Jiang, R., Zhou, F., Chen, Y., Wu, Y., & Ye, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1425 · Cited by 7 · see citing works
“1), text corpora are simply concatenated and divided into chunks of 2048 (Qiu et al., 2024) or 4096 (Guo et al., 2024) tokens, while the inherent semantic structure ( e.g. , catalogs of text-books) is discarded.”
PRobELM: Plausibility Ranking Evaluation for Language Models
Yuan, M., Whitehouse, C., Chamoun, E., Aly, R., & Vlachos, A. (2024) · arXiv.org · doi:10.48550/arxiv.2404.03818 · Cited by 9
“On the flip side, benchmarks such as TruthfulQA (Lin et al., 2022), FactScore (Min et al., 2023), and HaluEval (Li et al., 2023; Yu et al., 2023) measure the factuality of a model’s response exclusively.”
AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence
Alam, M. T., Bhusal, D., Ahmad, S., Rastogi, N., & Worth, P. J. (2025) · 2025 Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00072 · Cited by 8
“While general-purpose benchmarks such as GLUE [32], MMLU [33], and KOLA [34] broadly evaluate language model capabilities, they do not specifically target reasoning in cybersecurity domains.”
The ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11 · see citing works
“It is also necessary to prioritize code training techniques [74]to reduce carbon emissions [74, 94], and pay attention to environmental protection issues [121].”
Sun, K., Wang, R., Liu, H., & Sogaard, A. (2024) · arXiv.org · doi:10.48550/arxiv.2403.15250 · Cited by 8
“A primary issue is the narrow range of models typically assessed in multiple tasks — often several to 30 (Yu et al., 2023b; Yu et al., 2023a; Fu et al., 2023a; Jiang et al., 2023b; Huang et al., 2023), compared to the over 120000 models available, for instance, on Huggingface .”
BackgroundWhen does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks
Peng, H., Wang, X., Chen, J., Li, W., Qi, Y., Wang, Z., Wu, Z., Zeng, K., Xu, B., Hou, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.08993 · Cited by 7
Self-citationContext-Aware Reasoning On Parametric Knowledge for Inferring Causal Variables
Sheth, I., Abdelnabi, S., & Fritz, M. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.findings-emnlp.1194 · Cited by 7
“LLMs have demonstrated strong performance in internalizing knowledge (Sun et al., 2024; Yu et al., 2024) and reasoning-based tasks (Valmeekam et al., 2023; Guo et al., 2025), including causal discovery, where they infer pairwise causal relationships based on variable semantics (Kıcıman et al.,…”
KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models
Bai, Y., Feng, S., Balachandran, V., Tan, Z., Lou, S., He, T., & Tsvetkov, Y. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645623 · Cited by 5 · see citing works
“For example, factual editing [2, 6] requires the model to identify factual inconsistency and make corrections, rather than simply evaluating memorization; reasoning with structured knowledge [9, 64] examines the model’s ability to model knowledge in networks and graphs, instead of only probing knowledge at the atomic…”
BackgroundSelf-citationProtoEM: A Prototype-Enhanced Matching Framework for Event Relation Extraction
Hu, Z., Li, Z.-X., Xu, D., Bai, L., Jin, C., Jin, X.-L., Guo, J., & Cheng, X.-Q. (2023) · arXiv.org · doi:10.48550/arxiv.2309.12892 · Cited by 7
Cao, P., Chen, Y., Jin, Z.-R., Chen, Y., Liu, K., & Zhao, J. (2024) · Artificial Intelligence · doi:10.48550/arxiv.2411.17401 · Cited by 6
“Given the excellent knowledge capability, some researchers have constructed some evaluation benchmarks and attempted to use probing-based methods to assess the mastery of knowledge in LLMs [7], [11], [12].”
SadraeiJavaheri, M., Asgari, E., Mchardy, A., & Rabiee, H. (2024) · ENLSP · doi:10.48550/arxiv.2406.05279 · Cited by 6 · see citing works
“This process, referred to as “standardized overall scoring”, is described by Yu et al. (2023) and is employed in evaluating Large Language Models (LLMs).”
Qi, S., He, Y.-L., & Yuan, Z. (2024) · arXiv.org · doi:10.48550/arxiv.2404.12041 · Cited by 5
“Other than using the off-the-shelf methods, ExtEval (Zhang et al., 2023b) identifies five broad categories of unfaithfulness issues in extractive summarization that cannot be fully addressed by entailment models, with each category being assessed through a specific sub-metric.”
Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems
Zhang, Q.-F., Ruan, J.-H., Chen, A., Zeng, K., Chen, N., Tang, J., & Li, J. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2602.06319 · Cited by 5
“To address the imbalance in task difficulty across reasoning taxonomies, we follow prior work (Yuan et al., 2024; Yu et al., 2023) and apply a normalization procedure to reduce the influence of task complexity on model performance.”
Self-citationLLM-augmented entity alignment: an unsupervised and training-free framework
Long, M., Wang, J.-H., Ma, J., Zhou, J., & Chen, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108139 · Cited by 5
Educating LLMs like Human Students: Structure-aware Injection of Domain KnowledgePreprint
Liu, K., Chen, Z., Fu, Z.-H., Jiang, R.-X., Zhou, F., Chen, Y.-S., Wu, Y., & Ye, J.-P. (2024) · arXiv.org · doi:10.48550/arxiv.2407.16724 · Cited by 5 · see citing works
“1), text corpora are simply concatenated and divided into chunks of 2048 (Qiu et al., 2024) or 4096 (Guo et al., 2024) tokens, while the inherent semantic structure ( e.g. , catalogs of text-books) is discarded.”
Foundation Metrics: Quantifying Effectiveness of Healthcare Conversations powered by Generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Abad, Z. S. H., Thieme, A., Yang, Z., Wang, Y., Lin, B., Gevaert, O., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2309.12444 · Cited by 5
LLM-Driven Intrinsic Motivation for Sparse Reward Reinforcement Learning
Quadros, A., Silva, C., & Alves, R. C. O. (2025) · Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2025) · doi:10.48550/arxiv.2508.18420 · Cited by 4
“Due to the colossal amount of information with which these models are trained, these models end up being imbued with a certain amount of world knowledge [Yu et al. 2023].”
Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
Jiang, B.-W., Zhu, R.-C., Wu, J., Jiang, Z., He, Y.-F., Gao, J.-Y., Yu, J., Min, R., Wang, Y.-F., Yang, H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2505.16591 · Cited by 4
“It serves not only as the corpus for nearly all LLM pretraining (Chen et al. (2020)), but also as the foundation for constructing SFT data (Wang et al. (2024)) and evaluation datasets (Yu et al. (2023); He et al. (2024b;a)).”
Self-citationA goal-oriented document-grounded dialogue based on evidence generation
Song, Y. I., Fan, H., Liu, J., Liu, Y., Ye, X., & Ouyang, Y. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102378 · Cited by 3 · see citing works
Self-citationTrinity: A Modular Humanoid Robot AI System
Sun, J.-K., Zhang, Q., Han, G., Zhao, W., Yong, Z.-X., He, Y., Wang, J.-X., Cao, J.-H., Guo, Y.-J., & Xu, R.-J. (2025) · arXiv.org · doi:10.48550/arxiv.2503.08338 · Cited by 3
Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation
Silva, U. D., Samaraweera, D., Wanigathunga, S., Kariyawasam, K., Ranasinghe, K., Naseer, M., & Rodrigo, R. (2025) · Neural Networks · doi:10.48550/arxiv.2501.04696 · Cited by 3
“Modern LLMs, while limited to language modality, are known to contain knowledge regarding such obscure terms used across even some highly specialized domains [61].”
A Technical Taxonomy of LLM Agent Communication Protocols
Sander, L., Gidey, H. K., Lenz, A., & Knoll, A. (2026) · arXiv.org · doi:10.48550/arxiv.2606.19135 · Cited by 2
“Thanks to the LLM acting as the central controller, the agent has access to a vast amount of encoded knowledge embedded within the foundation model’s weights [4, 54].”
Kalian, A. D., Lee, J., Johannesson, S. P., Otte, L., Hogstrand, C., & Guo, M. (2025) · arXiv.org · doi:10.48550/arxiv.2506.20598 · Cited by 2
“Beyond this, they also demonstrate remarkable reasoning, planning, and broad world knowledge acquired during pre-training [22-25].”
Fan, Y., Li, R., Zhang, G., Shi, C., & Wang, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888877 · Cited by 2 · see citing works
“LLM hallucination evaluation benchmarks include TruthfulQA [26], FACTOR [27], HaluEval [28] and KoLA-KC [29].”
Self-citationWang, K.-D., Chen, T.-R., Hung, Y.-H., Ding, S.-Y., Wu, Y.-H., Wang, Y., Yang, C.-H. H., Peng, W.-C., & Hsieh, P.-C. (2025) · arXiv.org · doi:10.48550/arxiv.2502.20795 · Cited by 1
HeavyWater and SimplexWater: Distortion-Free LLM Watermarks for Low-Entropy Next-Token Predictions
Tsur, D., Long, C., Verdun, C. M., Hsu, H., Chen, C.-F., Permuter, H. H., Vithana, S., & Calmon, F. (2025) · arXiv.org · doi:10.48550/arxiv.2506.06409 · Cited by 1
Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability AnalysisPreprint
Sander, L., Pan, F., Zolfaghari, V., Schamschurko, A., Petrovic, N., & Knoll, A. (2026) · arXiv.org · doi:10.48550/arxiv.2607.27942 · Cited by 1 · see citing works
“The world knowledge embedded in LLMs [4], advanced reasoning and planning capabilities [5] and strong generalizability [6] make them powerful foundations for autonomous decision-making and qualify them as natural core controllers in modern agent systems [7].”
Fact-Checking with Large Language Models via Probabilistic Certainty and Consistency
Wang, H., Khalid, M., Wu, Q., Gao, J., & Cao, C. (2026) · arXiv.org · doi:10.48550/arxiv.2601.02574 · Cited by 1
“Since LLMs are pretrained on vast corpora, they encode substantial amounts of world knowledge in their parameters (Yu et al., 2023), enabling them to verify many claims without external retrieval.”
PIANIST: Learning Partially Observable World Models with LLMs for Multi-Agent Decision Making
Light, J., Xing, S.-X., Liu, Y.-Z., Chen, W., Cai, M., Chen, X., Wang, G.-Z., Cheng, W., Yue, Y.-S., & Hu, Z. (2024) · arXiv.org · doi:10.48550/arxiv.2411.15998 · Cited by 1
“They also possess extensive world knowledge [16], which has spurred increasing efforts to use LLMs for decision-making in text agents [17].”
Self-citationSalim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1 · see citing works
“Some popular benchmarks for LLMs include AGIEval, MMLU, Chatbot Arena, etc[28], [29], [30], [31], [32].”
Kardanova, E., Ivanova, A., Tarasova, K., Pashchenko, T., Tikhoniuk, A., Yusupova, E., Kasprzhak, A., Kuzminov, Y., Kruchinskaia, E., & Brun, I. (2024) · arXiv.org · doi:10.48550/arxiv.2411.00045 · Cited by 1 · see citing works
Qiang, Z.-C. (2025) · arXiv.org · doi:10.48550/arxiv.2503.21813
“There are also task-specific LLM hallucination benchmarks in dialogue summarisation [32], search engine augmentation [33], tool use [41], retrieval-augmented generation (RAG) [20], and knowledge graphs (KGs) [37].”
Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
Mitra, S., Neupane, S., Chakraborty, T., Tripathi, H., Mittal, S., Piplai, A., & Rahimi, S. (2026) · arXiv.org · doi:10.48550/arxiv.2607.18725
“Some, such as GLUE-X [15] and PromptBench [16], assess general capabilities including robustness to out-of-distribution and adversarial inputs , while KOLA [17] evaluates knowledge and reasoning.”
What Images Cannot Say: Language-Guided Olfactory Representation Learning
Tsonis, E., Wang, X., & Kalogeiton, V. (2026) · European Conference on Computer Vision · doi:10.48550/arxiv.2607.06402
“VLMs trained on large-scale data develop world knowledge that extends well beyond visual appearence, learning to reason about physical, acoustic, and semantic properties of scenes [27,60,69].”
Self-citationMultidimensional logical reasoning for event relation extraction
Qi, S., Zhang, J., Wei, X.-F., He, F., Yin, Z., Li, Q., & Ma, J. (2026) · International Conference on Machine Learning and Artificial Intelligence Applications · doi:10.1117/12.3110505
Robustness Evaluation of Tactics, Techniques, and Procedures Knowledge in Large Language Models
Chen, Y.-K., Lang, B., Xiao, N., Li, X.-Y., & Chen, R.-B. (2025) · IEEE International Conference on Systems, Man and Cybernetics · doi:10.1109/smc58881.2025.11343068
“To assess LLMs as knowledge bases, benchmarks like LAMA [12] and its successors (e.g., KoLA [13], Geo-LAMA [14]) have probed factual and conceptual knowledge via cloze-style queries.”
Design and Application of an Evaluation System for Large Language Models
Shen, Y., Cao, Z., & Chen, M. (2025) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai67447.2025.11412740
“KoLA [16] and Glue-X [17] evaluate LLM robustness against evolving knowledge—such as frequent Wikipedia updates—and out-of-distribution (OOD) data, respectively.”
Memory Capacity Assessment of Large Language Models: A Knowledge-Frequency Perspective
Zhou, L., Hou, X., & Liu, Y. (2025) · International Conference on Communication, Computing & Security · doi:10.1109/icccs65393.2025.11069850
“The frequency ranges including: 0 , [1 , 10) , [10 , 100) and [100 , inf ) .”
InfluentialSelf-citationLeveraging Large Language Models for Fuzzy String Matching in Political Science
Wang, Y. (2024) · arXiv.org · doi:10.48550/arxiv.2403.18218
Ebrahimzadeh, A., & Salili, S. M. (2026) · arXiv.org · doi:10.48550/arxiv.2601.02023
Measuring and Reducing LLM Hallucination without Gold-Standard Answers
Wei, J., Yao, Y., Ton, J.-F., Guo, H., Estornell, A., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.10412 · Cited by 10 · see citing works
“We measure hallucination as a continuous degree, which aligns with [16, 22, 42].”
MethodSelf-citationKnowTuning: Knowledge-aware Fine-tuning for Large Language Models
Lyu, Y., Yan, L., Wang, S., Shi, H., Yin, D., Ren, P., Chen, Z., de Rijke, M., & Ren, Z. (2024) · arXiv · doi:10.48550/arxiv.2402.11176 · Cited by 1 · see citing works
“In particular, they are prone to generating responses that are non-factual (Lee et al., 2022; Sun et al., 2023; Su et al., 2022; Wang et al., 2023b), incomplete (Singhal et al., 2022; Bian et al., 2023), or illogical (Chen et al., 2023c; Zhong et al., 2023; Kang et al., 2023).”
BackgroundInfluential
- Authors
- Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., Yao, Z., Zhang, X., Li, H., Li, C., Zhang, Z., Bai, Y., Liu, Y., Xin, A., Lin, N., Yun, K., Gong, L., Chen, J., Wu, Z., Qi, Y., Li, W., Guan, Y., Zeng, K., Qi, J., Jin, H., Liu, J., Gu, Y., Yao, Y., Ding, N., Hou, L., Liu, Z., Xu, B., Tang, J., & Li, J.
- Affiliations
- Tsinghua University (CN)
- ORCID
- Yu, Jifan
- Wang, Xiaozhi
- Tu, Shangqing
- Cao, Shulin
- Zhang-Li, Daniel
- Peng, Hao
- Yao, Zijun
- Zhang, Xiaohan
- Li, Hanming
- Li, Chunyang
- Zhang, Zheyuan
- Bai, Yushi
- Liu, Yantao
- Xin, Amy
- Lin, Nianyi
- Gong, Linlu
- Chen, Jianhui
- Wu, Zhili
- Qi, Yunjia
- Zeng, Kaisheng
- Jin, Hailong
- Liu, Jinxin
- Yao, Yuan
- Ding, Ning
- Hou, Lei
- Liu, Zhiyuan
- Xu, Bin
- Tang, Jie
- Li, Juanzi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.09296
- OpenAlex
- W4380994495
- Cited by
- 29
- References
- 85
- Citations by year
- 2025: 12 · 2024: 13 · 2023: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, world knowledge, benchmark design, generalization to unseen data, knowledge production
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
