CoChat
Sign Up
Cited by 2,842Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

A Survey on Evaluation of Large Language Models

Chang, Yupeng, Wang, Xu, Wang, Jindong, Wu, Yuan, Yang, Linyi, Zhu, Kaijie, Chen, Hao, Yi, Xiaoyuan +8 more·2024·ACM Transactions on Intelligent Systems and Technology
DOI 10.1145/3641289
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews evaluation of large language models along three dimensions: what to evaluate, where to evaluate, and how to evaluate. It argues that evaluation should be treated as an essential discipline and summarizes LLM success/failure cases, benchmarks, metrics, and future challenges.

Key findings

  • The survey finds no evidence that one evaluation protocol or benchmark is the most useful or successful; different benchmarks have different characteristics and focuses, and no single model performs best across all tasks.
  • LLMs succeed at text generation, language understanding tasks such as sentiment analysis and text classification, arithmetic and logical reasoning, temporal reasoning, machine translation, text generation, and question answering. They fail or struggle with NLI, event semantic similarity, basic phrase evaluation, abstract reasoning, complex contexts, non-Latin and low-resource languages, visual modality, toxic/biased outputs, credibility/hallucination, real-time information, and adversarial prompts.
  • The paper compiles 46 popular benchmarks and categorizes them into general language tasks, specific downstream tasks, and multi-modal tasks. It summarizes automatic evaluation metrics including accuracy (Exact Match, F1, ROUGE), calibration (Expected Calibration Error, AUC), fairness (Demographic Parity Difference, Equalized Odds Difference), and robustness (Attack Success Rate, Performance Drop Rate), plus human evaluation criteria: accuracy, relevance, fluency, transparency, safety, and human alignment.
  • Reported examples include GPT-4 and BingChat answering more than 80% of open questions correctly but still with a gap of over 15% to complete accuracy; GPT-4 scoring 117 on an emotional quotient test, surpassing 89% of human participants; GPT-4 achieving 76.4% versus GPT-3.5’s 46.8% in surgical clinical information; role-playing increasing generated toxicity up to 6x; and ChatGPT answering 83% at recognition level, 62% at comprehension, 27% at application, and 10% at highest cognitive complexity.

Methods

  • Design: a comprehensive literature survey of LLM evaluation, organized around “what to evaluate,” “where to evaluate,” and “how to evaluate,” covering tasks, datasets/benchmarks, and evaluation protocols.
  • Data or sample: synthesizes existing evaluation studies across NLP, robustness/ethics/biases/trustworthiness, social science, natural science and engineering, medical applications, agent applications, and other applications; compiles 46 popular benchmarks in Table 7 and maintains related open-source materials.
  • Main analysis: qualitative synthesis and tabulation of reported results, metrics, benchmarks, and protocols; the paper does not conduct new experiments.

Limitations

  • Stated: the authors may miss new papers or benchmarks because LLM evaluation is continuously developing; LLMs may become stronger and some described limitations may be mitigated, while new limitations may arise; conclusions depend on specific datasets; and there is no evidence that one evaluation protocol or benchmark is most useful.
  • Stated: the paper’s categorization is only one possible way to classify evaluation works, and several NLP areas intersect.
  • Evident from text: as a survey, it relies on results reported by cited papers rather than re-running experiments, so its conclusions inherit those papers’ datasets, protocols, and evaluation conditions; the benchmark list is explicitly not guaranteed to be exhaustive.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3641289

Cites

327 of 327 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

38 of 282 cited sources checked · 30 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    ChatGPT's performance on verbal insight problems is comparable to that of human participants.

    “Orrù et al. [147] demonstrated the potential of ChatGPT for solving verbal insight problems, as ChatGPT's performance was comparable to that of human participants.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [147] [147] Graziella Orrù, Andrea Piarulli, Ciro Conversano, and Angelo Gemignani. 2023. Human-like problem-solving abilities in large language models using ChatGPT. Frontiers in Artificial Intelligence 6 (2023). - source
  • Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT's average score in the student comprehensive assessment in primary care falls below the passing score.

    “Thirunavukarasu et al. [186] assessed ChatGPT's performance in primary care and found that its average score in the student comprehensive assessment falls below the passing score, indicating room for improvement.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [186] [186] Arun James Thirunavukarasu, Refaat Hassan, Shathar Mahmood, Rohan Sanghera, Kara Barzangi, Mohanned El Mukashfi, and Sachin Shah. 2023. Trialling a large language model (ChatGPT) in general practice with the Applied Knowledge Test: observational study demonstrating opportunities and limitations in primary care. JMIR Medical Education 9, 1 (2023), e46599. - source

      The Results report ChatGPT’s average score as 60.17%, below the recent mean AKT passing mark of 70.42%. But the Introduction identifies the AKT as an examination UK GPs must pass to complete their training, not a student comprehensive assessment; the claim misidentifies the assessment. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 3.5 Medical Applications · p. 18

    The presence of out-of-context information was lower compared to the correct answer in the NBME-Free-Step1 and NBME-Free-Step2 datasets.

    “However, the presence of out-of-context information was found to be lower compared to the correct answer in the NBME-Free-Step1 and NBME-Free-Step2 datasets.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [57] [57] Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does CHATGPT perform on the United States Medical Licensing Examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education 9, 1 (2023), e45312. - source
  • Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT achieves or approaches the passing threshold in USMLE exams with no tailored training.

    “Kung et al. [97] showed that ChatGPT achieves or approaches the passing threshold in these exams with no tailored training.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [97] [97] Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS digital health 2, 2 (2023), e0000198. - source
  • Premisecitation
    § 3.5 Medical Applications · p. 19

    GPT-4 consistently performs well across different subspecialties.

    “Notably, GPT-4 consistently performs well across different subspecialties, suggesting its capability to comprehend complex clinical information and enhance surgical education and training.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [143] [143] Namkee Oh, Gyu-Seong Choi, and Woo Yong Lee. 2023. ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models. Annals of Surgical Treatment and Research 104, 5 (2023), 269. - source
  • May combine two claims. 105
  • Classified as probably not a substantial claim. 103
  • The quote only partly states this claim. 3

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 7

    ChatGPT's sentiment analysis prediction performance is superior to traditional sentiment analysis methods and comes close to that of GPT-3.5.

    “ChatGPT's sentiment analysis prediction performance is superior to traditional sentiment analysis methods [129] and comes close to that of GPT-3.5 [159].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [129] [129] Alejandro Lopez-Lira and Yuehua Tang. 2023. Can chatgpt forecast stock price movements? Return predictability and large language models. arXiv preprint arXiv:2304.07619 (2023). -
    • cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 12

    The TruthfulQA dataset has become widely used for evaluating the factuality of LLMs.

    “This dataset has become widely used for evaluating the factuality of LLMs [89, 146, 192, 220].”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [89] [89] Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. ArXiv ab source

      Section 1.2 (p. 5) lists TruthfulQA among the paper’s evaluation tasks, and Section 1.3 (p. 7) identifies it as a benchmark for truthfulness, which the paper relates to factual accuracy. The paper establishes its use for this purpose but does not establish that it had become widely used. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [146] [146] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] - source
    • cites [192] [192] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). - source
    • cites [220] [220] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res. 2022 (2022). - source

      Section 3 (p. 3) uses TruthfulQA to evaluate whether language models answer questions truthfully, and §5.4 (p. 9) discusses results for GPT-3 and Gopher. These passages support its use as an evaluation benchmark, but do not establish that it had become widely used. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    LLMs internalize, spread, and potentially magnify harmful information and social biases existing in crawled training corpora.

    “LLMs have been found to internalize, spread, and potentially magnify harmful information existing in the crawled training corpora, usually, toxic languages, like offensiveness, hate speech, and insults [53], as well as social biases like stereotypes towards people with a particular demographic identity ( e.g. , gender, race, religion, occupation, and ideology) [175].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [53] [53] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020 . 3356-3369. - source
    • cites [175] [175] Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal Biases in Language Generation: Progress and Challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . 4275-4293. - source

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 16

    GPT-4 reached 60% accuracy on half of the categories of difficult high school competition problems, while intermediate algebra and precalculus can only be solved with a low accuracy rate of around 20%.

    “Wu et al. [225] evaluated the use of GPT-4 on difficult high school competition problems and GPT-4 reached 60% accuracy on half of the categories. Intermediate algebra and precalculus can only be solved with a low accuracy rate of around 20%.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [225] [225] Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. 2023. An Empirical Study on Challenging Math Problem Solving with GPT-4. arXiv preprint arXiv:2306.01337 (2023). - source

      The evaluation’s Table 1 reports approximately 60% accuracy in three of six categories for MathChat using GPT-4, not for GPT-4 under vanilla prompting; the latter is identified as showing GPT-4’s native capability and does not reach 60% in any category. The same passage explicitly says that Intermediate Algebra and Precalculus were solved at around 20% accuracy across the methods considered. The citing sentence omits the MathChat-method qualifier on its 60% result. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 15

    LLMs can generate coherent and seemingly factual text, but the information generated can include factual inaccuracies or statements ungrounded in reality, a phenomenon known as hallucination.

    “LLMs are capable of generating coherent and seemingly factual text. However, the information generated can include factual inaccuracies or statements ungrounded in reality, a phenomenon known as hallucination [163, 253].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [163] [163] Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. arXiv preprint arXiv:2309.05922 (2023). - source
    • cites [253] [253] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309.01219 [cs.CL] - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts.

    “Chervenak et al. [21] highlighted that while ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts, its limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] [21] Joseph Chervenak, Harry Lieman, Miranda Blanco-Breindel, and Sangita Jindal. 2023. The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertility and Sterility (2023). - source

  • May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 19

    GPT-3.5 achieved 46.8% overall accuracy and GPT-4 achieved 76.4% on understanding surgical clinical information, showing a significant performance difference.

    “The results indicate an overall accuracy of 46.8% for GPT-3.5 and 76.4% for GPT-4, demonstrating a significant performance difference between the two models.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [143] [143] Namkee Oh, Gyu-Seong Choi, and Woo Yong Lee. 2023. ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models. Annals of Surgical Treatment and Research 104, 5 (2023), 269. - source

  • May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 19

    LLMs demonstrate potential applications in medical assistance, including identifying gastrointestinal diseases, dementia diagnosis, accelerating COVID-19 literature evaluation, and overall healthcare.

    “In the field of medical assistance, LLMs demonstrate potential applications, including research on identifying gastrointestinal diseases [99], dementia diagnosis [217], accelerating the evaluation of COVID-19 literature [93], and their overall potential in healthcare [17].”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [99] [99] Adi Lahat, Eyal Shachar, Benjamin Avidan, Zina Shatz, Benjamin S Glicksberg, and Eyal Klang. 2023. Evaluating the use of large language model in identifying top research questions in gastroenterology. Scientific reports 13, 1 (2023), 4164. - source

      The Abstract and Discussion describe ChatGPT’s potential to identify research priorities and generate research questions in gastroenterology. They do not report using an LLM to identify gastrointestinal diseases, which is the application attributed to this source by citation [99]. The dementia, COVID-19, and broader healthcare examples have separate citations in the citing sentence. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [217] [217] Zhuo Wang, Rongzhen Li, Bowen Dong, Jie Wang, Xiuxing Li, Ning Liu, Chenhui Mao, Wei Zhang, Liling Dong, Jing Gao, et al. 2023. Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today. arXiv preprint arXiv:2306.01499 (2023). -

      Not checked in the free run.

    • cites [93] [93] Yousuf A Khan, Clarisse Hokia, Jennifer Xu, and Ben Ehlert. 2023. covLLM: Large Language Models for COVID-19 Biomedical Literature. arXiv preprint arXiv:2306.04926 (2023). -

      Not checked in the free run.

  • The quote only partly states this claim.

    Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT achieves varying accuracies across different datasets on USMLE Step 1 and Step 2 exam questions.

    “The results indicated that ChatGPT achieves varying accuracies across different datasets.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [57] [57] Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does CHATGPT perform on the United States Medical Licensing Examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education 9, 1 (2023), e45312. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT's limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.

    “Chervenak et al. [21] highlighted that while ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts, its limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] [21] Joseph Chervenak, Harry Lieman, Miranda Blanco-Breindel, and Sangita Jindal. 2023. The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertility and Sterility (2023). - source

  • May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT generated relatively accurate information for various medical queries, demonstrating its effectiveness in the field of medical queries to a certain extent.

    “ChatGPT generated relatively accurate information for various medical queries, including genetics [39], radiation oncology physics [73], biomedicine [81], and many other medical disciplines [63, 87, 169], demonstrating its effectiveness in the field of medical queries to a certain extent.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [39] [39] Dat Duong and Benjamin D Solomon. 2023. Analysis of large-language model versus human performance for genetics questions. European Journal of Human Genetics (2023), 1-3. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [73] [73] Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T Sio, Lisa A McGee, Jonathan B Ashman, Xiang Li, Tianming Liu, Jiajian Shen, et al. 2023. Evaluating large language models on a highly-specialized topic, radiation oncology physics. arXiv preprint arXiv:2304.01938 (2023). - source
    • cites [81] [81] Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023. Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers. arXiv preprint arXiv:2306.04504 (2023). - source

  • May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 18

    ChatGPT demonstrates high consistency and insight.

    “The model demonstrates high consistency and insight, indicating its potential to assist in medical education and clinical decision-making.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [97] [97] Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS digital health 2, 2 (2023), e0000198. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    ChatGPT can accurately assess student assignments and provide feedback on task completion.

    “It can accurately assess student assignments and provide feedback on task completion, thereby assisting in the development of student skills.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] [30] Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gasevic, and Guanliang Chen. 2023. Can large language models provide feedback to students? a case study on chatgpt. (2023). - source

      The Results on RQ3 report that ChatGPT generated task-level feedback for every report, and the Implications section suggests potential for developing learning skills. But the Implications section explicitly says ChatGPT could not reliably assess student performance compared with the instructor. The study does not establish that the feedback actually developed students’ skills. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    ChatGPT generates detailed, fluent, and coherent feedback that surpasses human teachers.

    “According to Dai et al. [30], ChatGPT demonstrates the ability to generate detailed, fluent, and coherent feedback that surpasses that of human teachers.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] [30] Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gasevic, and Guanliang Chen. 2023. Can large language models provide feedback to students? a case study on chatgpt. (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 21

    ChatGPT exhibits strong performance in recommender systems.

    “Dai et al. [29] suggest that ChatGPT exhibits strong performance in recommender systems.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [29] [29] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 24

    Dialogue CoT was introduced to assess in-depth dialogue, incorporating Explicit CoT and CoT.

    “To effectively assess in-depth dialogue, Wang et al. [205] introduced the Dialogue CoT, incorporating two efficient dialogue strategies: Explicit CoT and CoT.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [205] [205] Hongru Wang, Rui Wang, Fei Mi, Zezhong Wang, Ruifeng Xu, and Kam-Fai Wong. 2023. Chain-of-thought prompting for responding to in-depth dialogue questions with LLM. arXiv:2305.11792 [cs.CL] - source

      The contributions passage (page not available in our copy) says the paper proposes two dialogue CoT approaches and builds a benchmark for evaluating in-depth dialogue responses. But it names the approaches O-Cue CoT and M-Cue CoT, not Explicit CoT and CoT. Section 3.1 likewise lists O-Cue CoT and M-Cue CoT as the prompting schemes. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    GLUE-X is a novel attempt to create a unified benchmark aimed at evaluating robustness of NLP models in OOD scenarios.

    “GLUE-X [234] is a novel attempt to create a unified benchmark aimed at evaluating the robustness of NLP models in OOD scenarios.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [234] [234] Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2022. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073 (2022). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    BOSS is a benchmark collection for assessing out-of-distribution robustness in natural language processing tasks.

    “Yuan et al. [239] presents BOSS, a benchmark collection for assessing out-of-distribution robustness in natural language processing tasks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [239] [239] Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations. arXiv:2306.04618 [cs.CL] - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 23

    BIG-bench introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.

    “BIG-bench [182] introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [182] [182] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022). - source

  • May combine two claims.

    Premisecitation
    § 6.1.2 When can LLMs fail? · p. 30

    LLMs may exhibit social biases and toxicity during generation, resulting in biased outputs.

    “LLMs may exhibit social biases and toxicity [37, 53, 153] during the generation process, resulting in the production of biased outputs.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [37] [37] Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency . 862-872. - source
    • cites [53] [53] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020 . 3356-3369. - source
    • cites [153] [153] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 . 2086-2105. - source

      The Introduction says language models can reproduce social biases in language generation, and the Conclusion reports that QA model outputs can reinforce stereotypes. These passages support the biased-output claim, but the paper does not establish the separate claim about toxicity during generation. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 6.1.2 When can LLMs fail? · p. 31

    LLMs may manifest credibility deficits, potentially giving rise to fabricated information or erroneous facts within dialogues.

    “LLMs may manifest credibility deficits [201], potentially giving rise to fabricated information or erroneous facts within dialogues [163, 253].”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [201] [201] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. arXiv:2306.11698 [cs.CL] - source
    • cites [163] [163] Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. arXiv preprint arXiv:2309.05922 (2023). - source
    • cites [253] [253] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309.01219 [cs.CL] - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 INTRODUCTION · p. 2

    Large language models have incited substantial interest across both academic and industrial domains.

    “Recently, large language models (LLMs) have incited substantial interest across both academic and industrial domains [11, 219, 257].”
    • Also stated in: § A Survey on Evaluation of Large Language Models · p. 1 · plain
      “Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [11] [11] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). - source

      Not checked in the free run.

    • cites [219] [219] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). - source

      Not checked in the free run.

    • cites [257] [257] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning
    § 1 INTRODUCTION · p. 4

    Despite these efforts, a comprehensive overview capturing the entire gamut of evaluations is still lacking.

    “Despite these efforts, a comprehensive overview capturing the entire gamut of evaluations is still lacking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation
    § 3.1 Natural Language Processing Tasks · p. 7

    LLMs perform well on text classification and can handle text classification tasks in unconventional problem settings.

    “Overall, LLMs perform well on text classification and can even handle text classification tasks in unconventional problem settings as well.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    GPT-3.5 and Bard cannot distinguish between meaningful and nonsense phrases, consistently classifying highly nonsense phrases as meaningful.

    “Furthermore, GPT-3.5 and Bard cannot distinguish between meaningful and nonsense phrases, consistently classifying highly nonsense phrases as meaningful.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [166] [166] Nicholas Riccardi and Rutvik H Desai. 2023. The Two Word Test: A Semantic Benchmark for Large Language Models. arXiv preprint arXiv:2306.04610 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § 1 INTRODUCTION · p. 4

    The ongoing evolution of LLMs has presented novel aspects for evaluation, thereby challenging existing evaluation protocols.

    “Furthermore, the ongoing evolution of LLMs has also presented novel aspects for evaluation, thereby challenging existing evaluation protocols and reinforcing the need for thorough, multifaceted evaluation techniques.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given
    § 1 INTRODUCTION · p. 2

    As LLMs are becoming larger with more emergent abilities, existing evaluation protocols may not be enough to evaluate their capabilities and potential risks.

    “Finally, as LLMs are becoming larger with more emergent abilities, existing evaluation protocols may not be enough to evaluate their capabilities and potential risks.”
    • Also stated in: § 2.2 AI Model Evaluation · p. 6 · hedged
      “As LLMs are becoming more popular with even poorer interpretability, existing evaluation protocols may not be enough to evaluate the true capabilities of LLMs thoroughly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    GPT-4 shows significant improvements, but its performance is still significantly lower than that of humans.

    “GPT-4 shows significant improvements, but its performance is still significantly lower than that of humans.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [166] [166] Nicholas Riccardi and Rutvik H Desai. 2023. The Two Word Test: A Semantic Benchmark for Large Language Models. arXiv preprint arXiv:2306.04610 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § 1 INTRODUCTION · p. 2

    A significant takeaway from previous attempts is the paramount importance of AI evaluation, which serves as a critical tool to identify current system limitations and inform the design of more powerful models.

    “A significant takeaway from previous attempts is the paramount importance of AI evaluation, which serves as a critical tool to identify current system limitations and inform the design of more powerful models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation
    § 3.1 Natural Language Processing Tasks · p. 9

    Supervised models significantly outperform zero-shot models in terms of performance, highlighting that an increase in parameters does not necessarily guarantee a higher level of social knowledge in this particular scenario.

    “This statement demonstrates that supervised models significantly outperform zero-shot models in terms of performance, highlighting that an increase in parameters does not necessarily guarantee a higher level of social knowledge in this particular scenario.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 7

    ChatGPT excels in handling factual input that could be attributed to its RLHF training process in favoring human feedback.

    “They also found that ChatGPT excels in handling factual input that could be attributed to its RLHF training process in favoring human feedback.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.1 Natural Language Processing Tasks · p. 7

    The majority of evaluation research has been primarily focused on natural language tasks.

    “Consequently, the majority of evaluation research has been primarily focused on natural language tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 7

    ChatGPT can produce credibility ratings for a wide range of news outlets, and these ratings have a moderate correlation with those from human experts.

    “Yang and Menczer [233] found that ChatGPT can produce credibility ratings for a wide range of news outlets, and these ratings have a moderate correlation with those from human experts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [233] [233] Kai-Cheng Yang and Filippo Menczer. 2023. Large language models can rate news outlet credibility. arXiv preprint arXiv:2304.00228 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 10

    ChatGPT demonstrates comparable zero-shot summarization performance to text-davinci-002 but performs worse than GPT-3.5.

    “Specifically, ChatGPT demonstrates comparable zero-shot performance to the text-davinci-002 [6], but performs worse than GPT-3.5 [159].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

    • cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 3.1 Natural Language Processing Tasks · p. 9

    Most of the above reasoning conclusions are obtained for specific data sets.

    “It should be noted that most of the above conclusions are obtained for specific data sets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    ChatGPT lacks spatial reasoning ability but exhibits better temporal reasoning.

    “Meanwhile, ChatGPT also lacks spatial reasoning ability, but exhibits better temporal reasoning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    PaLM and Claude2 are the only two model families that achieve similar performance in multi-step reasoning, but they are still worse than the GPT model family.

    “For multi-step reasoning, Fu et al. [47] showed PaLM and Claude2 are the only two model families that achieve similar performance (but still worse than the GPT model family).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] [47] Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance. arXiv preprint arXiv:2305.17306 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation
    § 3.1 Natural Language Processing Tasks · p. 9

    ChatGPT's poor symbolic reasoning performance may be because it is prone to uncertain responses.

    “On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning.

    “ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    LLaMA-65B is the most robust open-source LLM to date and performs closely to code-davinci-002.

    “Moreover, LLaMA-65B is the most robust open-source LLMs to date, which performs closely to code-davinci-002.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] [47] Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance. arXiv preprint arXiv:2305.17306 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 10

    TNLG v2 (530B) achieved the highest summarization score in both scenarios, followed by OPT (175B) in second place.

    “Liang et al. [114] found that TNLG v2 (530B) [179] achieved the highest score in both scenarios, followed by OPT (175B) [247] in second place.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [114] [114] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisenone given
    § 3.1 Natural Language Processing Tasks · p. 10

    Claude and ChatGPT demonstrate competitive dialogue performance across evaluation dimensions, with Claude slightly outperforming ChatGPT in specific configurations.

    “When comparing the Claude and ChatGPT models, both models demonstrate competitive performance across different evaluation dimensions, with Claude slightly outperforming ChatGPT in specific configurations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 3.1 Natural Language Processing Tasks · p. 9

    LLMs show great potential in reasoning and a continuous improvement trend, but still face many challenges and limitations.

    “Overall, LLMs show great potential in reasoning and show a continuous improvement trend, but still face many challenges and limitations, requiring more in-depth research and optimization.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    Zero-shot InstructGPT and Codex are capable of complex medical reasoning tasks but still need further improvement.

    “In professional domain reasoning tasks, zero-shot InstructGPT and Codex are capable of complex medical reasoning tasks, but still need to be further improved [117].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [117] [117] Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143 (2022). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 10

    Fully fine-tuned task-specific models surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.

    “Research by Bang et al. [6] underscores that fully fine-tuned models tailored for specific tasks surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 9

    ChatGPT's performance is acceptable on causal and analogical reasoning but poor on multi-hop reasoning, similar to other LLMs' weakness on complex reasoning.

    “Finally, while the performance of ChatGPT is acceptable on causal and analogical reasoning, it performs poorly on multi-hop reasoning ability, which is similar to the weakness of other LLMs on complex reasoning [148].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [148] [148] Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. ThoughtSource: A central hub for large language model reasoning data. arXiv preprint arXiv:2301.11596 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 10

    The LMSYS-Chat-1M dataset encompasses up to one million samples.

    “Zheng et al. [259] have curated a comprehensive LLMs conversation dataset, LMSYS-Chat-1M, encompassing up to one million samples.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [259] [259] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. arXiv preprint arXiv:2309.11998 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Interpretationreasoning
    § 3.1 Natural Language Processing Tasks · p. 10

    LLMs perform satisfactorily in several translation tasks, but there is still room for improvement, especially in English-to-non-English translation.

    “In summary, while LLMs perform satisfactorily in several translation tasks, there is still room for improvement, e.g., enhancing the translation capability from English to non-English languages.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 10

    ChatGPT performs X-to-English translation well but lacks the ability to perform English-to-X translation.

    “The findings from Bang et al. [6] indicated that ChatGPT performs X → Eng translation well, but it still lacks the ability to perform Eng → X translation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § 3.1 Natural Language Processing Tasks · p. 10

    During contrastive testing, ChatGPT shows lower accuracy in comparison to traditional translation models.

    “During contrastive testing, ChatGPT shows lower accuracy in comparison to traditional translation models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 10

    Fine-tuned models such as Vícuna and ChatGPT exhibit near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin.

    “Fine-tuned models, such as Vícuna and ChatGPT, exhibit exceptional performance with near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin [5, 6].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [5] [5] Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2023. Benchmarking Foundation Models with Language-Model-as-an-Examiner. arXiv preprint arXiv:2306.04181 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.1 Natural Language Processing Tasks · p. 10

    ChatGPT's weaker performance on CommonsenseQA and Social IQA is attributed to its cautious nature of declining to answer when information is insufficient.

    “This can be attributed to ChatGPT's cautious nature, as it tends to decline to provide an answer when there is insufficient information available.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 11

    ChatGPT excels in assessing text quality from multiple angles even without reference texts, surpassing most existing automated metrics.

    “Chen et al. [20] revealed that ChatGPT excels in assessing text quality from multiple angles, even in the absence of reference texts, surpassing the performance of most existing automated metrics.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] [20] Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 3.1 Natural Language Processing Tasks · p. 10

    LLMs showcase flawless performance on QA tasks.

    “Overall, LLMs showcase flawless performance on QA tasks and hold the potential for further enhancing their proficiency in social, event, and temporal commonsense knowledge in the future.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given
    § 3.1 Natural Language Processing Tasks · p. 10

    GPT-4 demonstrates a robust capability in explaining discourse knowledge, even though it may occasionally select incorrect translation candidates.

    “However, GPT-4 demonstrates a robust capability in explaining discourse knowledge, even though it may occasionally select incorrect translation candidates.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § 3.1 Natural Language Processing Tasks · p. 11

    There are numerous challenges and ample opportunities for enhancement in multilingual tasks for LLMs.

    “The aforementioned demonstrates that there are numerous challenges and ample opportunities for enhancement in multilingual tasks for LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    LLMs demonstrate a propensity for progressive views and an ENFJ personality type on questionnaires.

    “demonstrating a propensity for progressive views and an ENFJ personality type.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [65] [65] Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768 (2023). - source

      Not checked in the free run.

    • cites [167] [167] Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, and Markus Pauly. 2023. The Self-Perception and Political Biases of ChatGPT. arXiv preprint arXiv:2304.07333 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    The CHBias dataset discovers bias risks in pre-trained models.

    “[256] proposes a Chinese conversational bias evaluation dataset CHBias, discovers bias risks in pre-trained models, and explores debiasing methods.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [256] [256] Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, and Mykola Pechenizkiy. 2023. CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models. arXiv:2305.11262 [cs.CL] - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    Existing LMs have potential in ethical judgment but still need improvement.

    “The study conducted by [69] reveals that existing LMs have potential in ethical judgment, but still need improvement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] [69] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275 (2020). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 12

    GPT-4 and BingChat can provide correct answers for more than 80% of questions, with a remaining gap of over 15% to achieve complete accuracy.

    “The results of the study indicated that while GPT-4 and BingChat can provide correct answers for more than 80% of the questions, there is still a remaining gap of over 15% to achieve complete accuracy.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [204] [204] Cunxiang Wang, Sirui Cheng, Zhikun Xu, Bowen Ding, Yidong Wang, and Yue Zhang. 2023. Evaluating open question answering evaluation. arXiv preprint arXiv:2305.12421 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • The quote only partly states this claim.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 12

    An information-theory metric for assessing knowledge inclusion in LLMs surpasses traditional ranking methods by achieving an accuracy improvement of over 30%.

    “The study demonstrated that this approach surpasses traditional ranking methods by achieving an accuracy improvement of over 30%.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [156] [156] Pouya Pezeshkpour. 2023. Measuring and Modifying Factual Knowledge in Large Language Models. arXiv preprint arXiv:2306.06264 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 12

    Current factual consistency evaluation methods lack a unified comparison framework, and related scores have limited reference value compared to binary labels.

    “they conducted a review of current factual consistency evaluation methods and highlighted the absence of a unified comparison framework and the limited reference value of related scores compared to binary labels.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] [74] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991 (2022). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    GPT-4 often showcases improved trustworthiness over GPT-3.5 in standard evaluations, but it is simultaneously more susceptible to attacks.

    “They revealed that while GPT-4 often showcases improved trustworthiness over GPT-3.5 in standard evaluations, it is simultaneously more susceptible to attacks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [201] [201] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. arXiv:2306.11698 [cs.CL] - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    Role-playing increases generated toxicity up to 6x and causes biased toxicity toward specific entities.

    “Deshpande et al. [35] introduced role-playing into the model and observed an increase in generated toxicity up to 6x. Furthermore, such role-playing also caused biased toxicity towards specific entities.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] [35] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    LLMs with enhanced cognitive abilities can avoid common human intuitions and cognitive errors, demonstrating super-rational performance.

    “They found that these models can avoid common human intuitions and cognitive errors, demonstrating super-rational performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62] [62] Thilo Hagendorff and Sarah Fabi. 2023. Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models - and Disappeared in GPT-4. arXiv:2306.07622 [cs.CL] - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1 Natural Language Processing Tasks · p. 12

    Fact evaluation methods founded on natural language inference and question generation answering exhibit superior performance and can complement each other.

    “The research discovered that fact evaluation methods founded on natural language inference and question generation answering exhibit superior performance and can complement each other.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [74] [74] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991 (2022). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 14

    ChatGPT still exhibits noxious content to some extent.

    “finding that it still exhibits noxious content to some extend.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [266] [266] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning
    § 3.4 Natural Science and Engineering · p. 16

    GPT-4 outperforms ChatGPT by achieving a significant increase in accuracy of 10 percentage points and a reduction in relative error by 50%.

    “GPT-4 outperforms ChatGPT by achieving a significant increase in accuracy of 10 percentage points and a reduction in relative error by 50%, due to its superior division and trigonometry abilities, proper understanding of irrational numbers, and consistent step-by-step calculation of long expressions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 3.4 Natural Science and Engineering · p. 17

    ChatGPT's performance worsens as task difficulty increases, with accuracy of 83% at recognition, 62% at comprehension, 27% at application, and 10% at the highest cognitive complexity level.

    “showed that ChatGPT's performance worsens as task difficulty increases: it correctly answered 83% of the questions at the recognition level, 62% at the comprehension level, 27% at the application level, and only 10% at the highest cognitive complexity level.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.3 Social Science · p. 15

    LLMs have several problems in legal case judgment summarization, including incomplete sentences and words, meaningless sentence merge, and inconsistent and hallucinated information.

    “LLMs have several problems, including incomplete sentences and words, meaningless sentences merge, and more serious errors such as inconsistent and hallucinated information [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] [34] Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2023. How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization? arXiv preprint arXiv:2306.01248 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 16

    On complex and challenging mathematical problems, GPT-3 demonstrates nearly random performance, GPT-3.5 shows improvement, and GPT-4 performs best.

    “Specifically, GPT-3 demonstrates nearly random performance, while GPT-3.5 shows improvement, and GPT-4 performs the best [3].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3] [3] Daman Arora, Himanshu Gaurav Singh, et al. 2023. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. arXiv preprint arXiv:2305.15074 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 16

    Peak LLM performance on mathematics remains relatively low compared to experts, and these models lack the capability to engage in mathematical research.

    “Despite the advancements made in the new models, it is important to note that the peak performance remains relatively low compared to that of experts and these models lack the capability to engage in mathematical research [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] [15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 16

    LLMs exhibit competence in handling decimal numbers, negative numbers, and irrational numbers.

    “On the other hand, LLMs exhibit competence in handling decimal numbers, negative numbers, and irrational numbers [241].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [241] [241] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 16

    Most LLMs demonstrate proficiency in addition and subtraction, and possess some capability in multiplication.

    “For fundamental mathematical problems, most large language models (LLMs) demonstrate proficiency in addition and subtraction, and possess some capability in multiplication.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [241] [241] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015 (2023). - source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 3.3 Social Science · p. 15

    Further improvement is necessary for LLMs to be useful for case judgment summarization by legal experts.

    “The results showed that further improvement is necessary for LLMs to be useful for case judgment summarization by legal experts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.3 Social Science · p. 15

    LLMs can greatly enhance the traditional CSS research pipeline but cannot completely replace it.

    “In summary, while LLMs can greatly enhance the traditional CSS research pipeline, they cannot completely replace it.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 3.3 Social Science · p. 15

    LLMs can generate meaningful responses regarding political ideology and significantly improve text-as-data methods in social science.

    “found that LLMs can generate meaningful responses regarding political ideology and significantly improve text-as-data methods in social science.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [224] [224] Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing. 2023. Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting. arXiv preprint arXiv:2303.12057 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.3 Social Science · p. 15

    In CSS generation tasks, LLMs often produce explanations that surpass the quality of gold references provided by crowd workers.

    “When it comes to generation tasks, LLMs often produce explanations that surpass the quality of gold references provided by crowd workers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [269] [269] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:2305.03514 (2023). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims. The quote only partly states this claim.

    Interpretationnone given
    § 3.3 Social Science · p. 15

    LLMs benefit social science tasks and improve work efficiency, but existing LLMs cannot completely replace human professionals in this domain.

    “In conclusion, the utilization of LLMs has significantly benefited individuals in addressing social science-related tasks, leading to improved work efficiency.”
    • Also stated in: § 3.3 Social Science · p. 15 · plain
      “However, it is crucial to acknowledge that existing LLMs cannot completely replace human professionals in this domain.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Robustness, Ethic, Bias, and Trustworthiness · p. 15

    LRV-Instruction effectively alleviates illusions in LLMs.

    “experimental results demonstrated that LRV-Instruction effectively alleviates illusions in LLMs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [123] [123] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning. arXiv:2306.14565 [cs.CV] - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 16

    LLMs face challenges with division, exponentiation, trigonometry functions, and logarithm functions.

    “However, they face challenges when it comes to division, exponentiation, trigonometry functions, and logarithm functions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [241] [241] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.3 Social Science · p. 15

    LLMs, particularly when combined with prompting enhancements and correct legal texts, could perform better but not yet at expert tax lawyer levels.

    “Nay et al. [139] indicated that LLMs, particularly when combined with prompting enhancements and the correct legal texts, could perform better but not yet at expert tax lawyer levels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [139] [139] John J Nay, David Karamardian, Sarah B Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H Choi, and Jungo Kasai. 2023. Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence. arXiv preprint arXiv:2306.07075 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.3 Social Science · p. 15

    LLMs achieve the best performance on misinformation, stance, and emotion classification.

    “Conversely, LLMs achieve the best performance on misinformation, stance, and emotion classification.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [269] [269] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:2305.03514 (2023). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    ChatGPT surpasses highly capable college students in dynamic programming, greedy algorithm, and search, but struggles in data structure, tree, and graph theory.

    “Specifically, ChatGPT has developed superior skills in dynamic programming, greedy algorithm, and search, surpassing highly capable college students, but it struggles in data structure, tree, and graph theory.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [265] [265] Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. 2023. Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective. arXiv preprint arXiv:2306.10512 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given
    § 3.4 Natural Science and Engineering · p. 17

    LLMs can handle simple engineering tasks, but they perform poorly on complex engineering tasks.

    “It turns out that LLMs can handle simple engineering tasks, but they perform poorly on complex engineering tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.5 Medical Applications · p. 19

    GPT-4 has the capability to comprehend complex clinical information and enhance surgical education and training.

    “Notably, GPT-4 consistently performs well across different subspecialties, suggesting its capability to comprehend complex clinical information and enhance surgical education and training.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    ChatGPT generally performs well in software engineering tasks and provides detailed responses, often surpassing both human expert output and SOTA output.

    “In software engineering tasks, ChatGPT generally performs well and provides detailed responses, often surpassing both human expert output and SOTA output.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [181] [181] Giriprasad Sridhara, Sourav Mazumdar, et al. 2023. ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks. arXiv preprint arXiv:2305.16837 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Interpretationreasoning
    § 3.5 Medical Applications · p. 18

    ChatGPT has potential to assist in medical education and clinical decision-making.

    “The model demonstrates high consistency and insight, indicating its potential to assist in medical education and clinical decision-making.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    ChatGPT fails to provide accurate answers for code vulnerability detection and information retrieval-based test prioritization, making it unsuitable for these specific tasks.

    “However, for certain tasks such as code vulnerability detection and information retrieval-based test prioritization, the current version of ChatGPT fails to provide accurate answers, rendering it unsuitable for these specific tasks [181].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [181] [181] Giriprasad Sridhara, Sourav Mazumdar, et al. 2023. ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks. arXiv preprint arXiv:2305.16837 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 19

    ChatGPT effectively translates radiology reports into accessible language and provides general recommendations.

    “The findings demonstrate that ChatGPT effectively translates radiology reports into accessible language and provides general recommendations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [131] [131] Qing Lyu, Josh Tan, Mike E Zapadka, Janardhana Ponnatapuram, Chuang Niu, Ge Wang, and Christopher T Whitlow. 2023. Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential. arXiv preprint arXiv:2303.09038 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    Chemistry problems have lower inference complexity than physics problems in this setting.

    “[3] showed that LLMs perform worse on physics problems than chemistry problems, probably because chemistry problems have lower inference complexity than physics problems in this setting.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3] [3] Daman Arora, Himanshu Gaurav Singh, et al. 2023. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. arXiv preprint arXiv:2305.15074 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.5 Medical Applications · p. 18

    Answers generated by ChatGPT are more context-aware with better deductive reasoning abilities compared to Google search results.

    “Moreover, Sharma et al. [173] found that answers generated by ChatGPT are more context-aware with better deductive reasoning abilities compared to Google search results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [173] [173] Prabin Sharma, Kisan Thapa, Prastab Dhakal, Mala Deep Upadhaya, Santosh Adhikari, and Salik Ram Khanal. 2023. Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education. arXiv preprint arXiv:2307.00112 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    LLMs' plan generalization capabilities appear to be limited.

    “Moreover, it explored the capability of LLMs for plan generalization and found that their generalization capabilities appear to be limited.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [150] [150] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia. 2023. Understanding the Capabilities of Large Language Models for Automated Planning. arXiv preprint arXiv:2305.16151 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    GPT-4 demonstrates an advanced ability to generate code based on given instructions, comprehend existing code, reason about code execution, simulate the impact of instructions, articulate outcomes in natural language, and execute pseudocode effectively.

    “GPT-4 demonstrates an advanced ability to generate code based on given instructions, comprehend existing code, reason about code execution, simulate the impact of instructions, articulate outcomes in natural language, and execute pseudocode effectively [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] [15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.4 Natural Science and Engineering · p. 17

    Fine-tuned CodeT5 performs best across all considered domains, with the shortest inference time.

    “Pallagani et al. [150] demonstrated that the fine-tuned CodeT5 [214] performs the best across all considered domains, with the shortest inference time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [150] [150] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia. 2023. Understanding the Capabilities of Large Language Models for Automated Planning. arXiv preprint arXiv:2305.16151 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Interpretationreasoning
    § 3.7 Other Applications · p. 21

    Incorporating LLMs into recommendation pipelines can offer more accurate and personalized recommendations, improving user experience and overall recommendation quality.

    “By incorporating LLMs into recommendation pipelines, these systems can offer more accurate and personalized recommendations, thereby improving user experience and overall recommendation quality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.5 Medical Applications · p. 19

    ChatGPT's quality has shown improvement compared to GPT-4.

    “Furthermore, the quality of ChatGPT has shown improvement compared to GPT-4.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [131] [131] Qing Lyu, Josh Tan, Mike E Zapadka, Janardhana Ponnatapuram, Chuang Niu, Ge Wang, and Christopher T Whitlow. 2023. Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential. arXiv preprint arXiv:2303.09038 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.6 Agent Applications · p. 19

    Understanding when and how to utilize external symbolic tools is important, and this knowledge is dependent on the capabilities of LLMs.

    “The study by MRKL Karpas et al. [90] emphasized the importance of understanding when and how to utilize external symbolic tools, as this knowledge is dependent on the capabilities of LLMs, particularly when these tools can reliably perform functions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et al. 2022. MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.00445 (2022). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.6 Agent Applications · p. 19

    Toolformer employs a training approach to determine optimal usage of specific APIs and integrates results into subsequent token predictions.

    “Toolformer employs a training approach to determine the optimal usage of specific APIs and integrates the obtained results into subsequent token predictions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [172] [172] TimoSchick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 21

    Safdari et al. proposed a comprehensive approach for psychometric testing of personality traits in LLM-generated text.

    “Safdari et al. [168] proposed a comprehensive approach to conduct effective psychometric testing for the personality traits in the text generated by LLMs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [168] [168] Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality Traits in Large Language Models. arXiv preprint arXiv:2307.00184 (2023). -

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 21

    Most LLMs achieve above-average scores in EQ, with GPT-4 scoring 117, surpassing 89% of human participants.

    “The results showed that most LLMs achieve above-average scores in emotional quotient (EQ), with GPT-4 scoring 117, surpassing 89% of human participants.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [212] [212] Xuena Wang, Xueting Li, Zi Yin, Yue Wu, and Liu Jia. 2023. Emotional Intelligence of Large Language Models. arXiv:2307.09042 [cs.AI] - source

  • May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    Guided ChatGPT and GPT-4 exhibit competitive performance on popular benchmark tests, even outperforming supervised methods.

    “Experimental results demonstrate that guided ChatGPT and GPT-4 exhibit competitive performance on popular benchmark tests, even outperforming supervised methods.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [183] [183] Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. arXiv preprint arXiv:2304.09542 (2023). - source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § 3.7 Other Applications · p. 20

    LLMs show promise in addressing program logic issues, but challenges remain in output formatting.

    “In conclusion, the utilization of LLMs shows promise in addressing program logic issues, although challenges remain in achieving proficiency in output formatting.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.6 Agent Applications · p. 19

    ToolLLM provides a comprehensive framework to equip open-source LLMs with tool use capabilities.

    “ToolLLM [161] provides a comprehensive framework to equip open-source large language models with tool use capabilities.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [161] [161] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789 [cs.AI] - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 20

    GPT-4 achieved a score of 8.33.

    “Subsequently, the evaluation was conducted using GPT-4, and it achieved a score of 8.33.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] [32] Joost CF de Winter. 2023. Can ChatGPT pass high school exams on English language comprehension. Researchgate. Preprint (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 21

    ChatGPT shows promise in addressing the cold-start problem and providing interpretable recommendations.

    “Furthermore, ChatGPT shows promise in addressing the cold-start problem and providing interpretable recommendations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] [29] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 21

    Davinci003 as a chatbot shows variations in consistency of answers, despite exhibiting prosocial characteristics.

    “Research conducted by Bodroza et al. [10] investigated the personality features of using Davinci003 as a chatbot and found variations in the consistency of its answers, despite exhibiting prosocial characteristics.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] [10] Bojana Bodroza, Bojana M Dinic, and Ljubisa Bojic. 2023. Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3's personality instruments results. arXiv preprint arXiv:2306.04308 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    ChatGPT achieves an average of 71.8% correctness, comparable to the average score of all participating students.

    “de Winter [32] showed that ChatGPT achieves an average of 71.8% correctness, which is comparable to the average score of all participating students.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] [32] Joost CF de Winter. 2023. Can ChatGPT pass high school exams on English language comprehension. Researchgate. Preprint (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 20

    ChatGPT's responses may lack novelty or insightful perspectives regarding teaching improvement.

    “However, ChatGPT's responses may lack novelty or insightful perspectives regarding teaching improvement [210].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [210] [210] Rose E Wang and Dorottya Demszky. 2023. Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction. arXiv preprint arXiv:2306.03090 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 21

    ChatGPT generates unfair recommendations.

    “Recent research by Zhang et al. [246] has highlighted the issue of unfair recommendations generated by ChatGPT.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [246] [246] Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation. arXiv preprint arXiv:2305.07609 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.6 Agent Applications · p. 19

    HuggingGPT leverages LLMs to connect various AI models to address AI tasks.

    “Furthermore, Shen et al. [174] proposed the HuggingGPT framework, which leverages LLMs to connect various AI models within the machine learning community (such as Hugging Face), aiming to address AI tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [174] [174] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.6 Agent Applications · p. 19

    KOSMOS-1 is capable of understanding general patterns, following instructions, and learning based on context.

    “Huang et al. [77] introduced KOSMOS-1, which is capable of understanding general patterns, following instructions, and learning based on context.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 20

    Leveraging bootstrapping that combines randomness via the 'temperature' parameter is effective in diagnosing incorrect answers.

    “Furthermore, this evaluation showed the effectiveness of leveraging bootstrapping that combines randomness via the 'temperature' parameter in diagnosing incorrect answers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] [32] Joost CF de Winter. 2023. Can ChatGPT pass high school exams on English language comprehension. Researchgate. Preprint (2023). - source

  • May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    LLMs can successfully identify at least one actual problem in student code, though misjudgments occur.

    “Additionally, the study conducted by Hellas et al. [67] revealed that LLMs can successfully identify at least one actual problem in student code, although instances of misjudgment are also observed.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] [67] Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests. arXiv preprint arXiv:2306.05715 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 21

    MoRec and TCF show promising potential in recommendation systems.

    “Moreover, the research by Yuan et al. [240] and Li et al. [110] demonstrated the promising potential of the modality-based recommendation model (MoRec) and text-based collaborative filtering (TCF) in recommendation systems.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [240] [240] Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023. Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited. arXiv:2303.13835 [cs.IR] - source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [110] [110] Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. 2023. Exploring the Upper Limits of TextBased Collaborative Filtering Using Large Language Models: Discoveries and Insights. arXiv preprint arXiv:2305.11700 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 21

    Many language models perform unreliably in self-assessment tests and exhibit inherent biases.

    “Song et al. [180] examined the manifestation of personality in language models and discovered that many models perform unreliably in self-assessment tests and exhibit inherent biases.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [180] [180] Xiaoyang Song, Akshat Gupta, Kiyan Mohebbizadeh, Shujie Hu, and Anant Singh. 2023. Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs. arXiv preprint arXiv:2305.14693 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.7 Other Applications · p. 20

    LLMs may generate errors similar to those made by students.

    “It is important to note that while these models can provide valuable insights, they may still generate errors similar to those made by students.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    The ChatGPT group spent less time on all tasks, and the difference between the two groups is not significant.

    “The results show that the ChatGPT group spent less time on all tasks and the difference between these two groups is not significant.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [232] [232] Ruiyun Xu, Yue Feng, and Hailiang Chen. 2023. ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience. arXiv preprint arXiv:2307.01135 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 20

    GPT-3.5 can solve MIT math and EECS exams, with GPT-4 achieving better performance.

    “Zhang et al. [248] claimed that GPT-3.5 can solve MIT math and EECS exams with GPT-4 achieving better performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [248] [248] Sarah J Zhang, Samuel Florin, Ariel N Lee, Eamon Niknafs, Andrei Marginean, Annie Wang, Keith Tyser, Zad Chin, Yann Hicke, Nikhil Singh, et al. 2023. Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models. arXiv preprint arXiv:2306.08997 (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 21

    Listwise ranking strikes the best balance between cost and performance.

    “The use of listwise ranking is found to strike the best balance between cost and performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] [29] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    PandaLM is a discriminative large-scale language model designed to differentiate among multiple high-proficiency LLMs through training.

    “PandaLM [216] is introduced as a discriminative large-scale language model specifically designed to differentiate among multiple high-proficiency LLMs through training.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [216] [216] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. arXiv preprint arXiv:2306.05087 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.7 Other Applications · p. 21

    ChatGPT demonstrates impressive capabilities in NLP tasks but falls short in generating humorous responses.

    “They found that while ChatGPT demonstrates impressive capabilities in NLP tasks, it falls short in generating humorous responses.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [84] [84] Sophie Jentzsch and Kristian Kersting. 2023. ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models. arXiv preprint arXiv:2306.04563 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    MMLU provides a comprehensive suite of tests for assessing text models in multi-task contexts.

    “MMLU [70] provides a comprehensive suite of tests for assessing text models in multi-task contexts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [70] [70] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    AGIEval serves as a dedicated evaluation framework for assessing performance of foundation models in human-centric standardized exams.

    “AGIEval [262], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [262] [262] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 24

    MultiMedQA is a medical QA benchmark focusing on medical examinations, medical research, and consumer healthcare questions.

    “MultiMedQA [177] is a medical QA benchmark that focuses on medical examinations, medical research, and consumer healthcare questions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [177] [177] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138 (2022). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    AlpacaEval is an automated evaluation benchmark focusing on assessing performance of LLMs across various natural language processing tasks.

    “AlpacaEval [112] stands as an automated evaluation benchmark, which places its focus on assessing the performance of LLMs across various natural language processing tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [112] [112] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsulab/alpaca\_eval. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Benchmarks for General Tasks · p. 24

    PromptBench centers on the importance of prompt engineering in fine-tuning LLMs.

    “PromptBench [264] centers on the importance of prompt engineering in fine-tuning LLMs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [264] [264] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arXiv preprint arXiv:2306.04528 (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.7 Other Applications · p. 21

    LLMs face difficulties in capturing the subtleties and context-dependent nature of humor.

    “This study emphasizes the importance of humor in human communication and the difficulties that LLMs face in capturing the subtleties and context-dependent nature of humor.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [84] [84] Sophie Jentzsch and Kristian Kersting. 2023. ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models. arXiv preprint arXiv:2306.04563 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 24

    FRESHQA was introduced to assess ability of LLMs in dynamic QA about current world knowledge.

    “To assess the ability of LLMs in dynamic QA about current world knowledge, Vu et al. [198] introduced FRESHQA.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [198] [198] Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. arXiv:2310.03214 [cs.CL] -

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    CELLO was introduced to assess LLMs' aptitude in understanding complex instructions.

    “to assess LLMs' aptitude in understanding complex instructions, He et al. [66] have introduced CELLO.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [66] [66] Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang, Yuncheng Huang, et al. 2023. Can Large Language Models Understand Real-World Complex Instructions? arXiv preprint arXiv:2309.09150 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    ARB focuses on probing performance of LLMs in advanced reasoning tasks spanning multiple domains.

    “ARB, as presented by Sawada et al. [171], focuses on probing the performance of LLMs in advanced reasoning tasks spanning multiple domains.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [171] [171] Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. ARB: Advanced Reasoning Benchmark for Large Language Models. arXiv:2307.13692 [cs.CL] - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    CMB is a medical evaluation benchmark rooted in Chinese language and culture, addressing potential inconsistency in local context from relying solely on English-based medical assessments.

    “Wang et al. [211] introduced CMB, a medical evaluation benchmark rooted in the Chinese language and culture. It addresses the potential inconsistency in the local context that may arise from relying solely on English-based medical assessments.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [211] [211] Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. 2023. CMB: A Comprehensive Medical Benchmark in Chinese. arXiv preprint arXiv:2308.08833 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    MATH concentrates on assessing reasoning and problem-solving proficiencies of AI models in mathematics.

    “MATH [72] concentrates on assessing reasoning and problem-solving proficiencies of AI models within the domain of mathematics.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [72] [72] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    Choice-75 was introduced to evaluate daily decision-making capabilities of intelligent systems.

    “To evaluate the daily decision-making capabilities of intelligent systems, Hou et al. [75] introduced Choice-75.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75] [75] Zhaoyi Joey Hou, Li Zhang, and Chris Callison-Burch. 2023. Choice-75: A Dataset on Decision Branching in Script Learning. arXiv preprint arXiv:2309.11737 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    C-Eval is the first extensive benchmark to assess advanced knowledge and reasoning capabilities of foundation models in Chinese.

    “C-Eval [78], which is the first extensive benchmark to assess the advanced knowledge and reasoning capabilities of foundation models in Chinese.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [78] [78] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    GAOKAO-Bench provides a comprehensive evaluation benchmark for gauging proficiency of LLMs in intricate and context-specific tasks using Chinese Gaokao examination questions.

    “GAOKAO-Bench [245] provides a comprehensive evaluation benchmark for gauging the proficiency of large language models in intricate and context-specific tasks, utilizing questions sourced from the Chinese Gaokao examination.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [245] [245] Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2023. Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning. arXiv preprint arXiv:2306.02408 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    SOCKET is an NLP benchmark designed to evaluate performance of LLMs in learning and recognizing social knowledge concepts.

    “SOCKET [23] serves as an NLP benchmark designed to evaluate the performance of LLMs in learning and recognizing social knowledge concepts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] [23] Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023. Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark. arXiv preprint arXiv:2305.14938 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    UHGEval is a benchmark specifically designed to evaluate performance of Chinese LLMs in text generation without being constrained by hallucination-related limitations.

    “[116] has developed UHGEval, a benchmark specifically designed to evaluate the performance of Chinese LLMs in text generation without being constrained by hallucination-related limitations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [116] [116] Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Zhaohui Wy, Dawei He, Peng Cheng, Zhonghao Wang, et al. 2023. UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation. arXiv preprint arXiv:2311.15296 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    CVALUES introduces a humanistic evaluation benchmark to assess alignment of LLMs with safety and responsibility standards.

    “CVALUES [230] introduces a humanistic evaluation benchmark to assess the alignment of LLMs with safety and responsibility standards.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [230] [230] Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. 2023. CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility. arXiv:2307.09705 [cs.CL] - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    CMMLU is a comprehensive Chinese proficiency standard and evaluates performance of 18 LLMs across various academic disciplines.

    “Li et al. [108] introduces CMMLU as a comprehensive Chinese proficiency standard and evaluates the performance of 18 LLMs across various academic disciplines.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [108] [108] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    CUAD is an expert-annotated, domain-specific legal contract review dataset that presents a challenging research benchmark and potential for enhancing deep learning models' performance in contract understanding tasks.

    “CUAD [71] is an expert-annotated, domain-specific legal contract review dataset that presents a challenging research benchmark and potential for enhancing deep learning models' performance in contract understanding tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [71] [71] Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268 (2021). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    MMBench was introduced to robustly evaluate large-scale vision-language models, comprising a comprehensive dataset and employing a CircularEval assessment method.

    “Liu et al. [126] introduced MMBench, which comprises a comprehensive dataset and employs a CircularEval assessment method.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [126] [126] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023. MMBench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281 [cs.CV] - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    MINT was proposed to evaluate LLMs in multi-turn interactions, utilizing tools and natural language feedback.

    “To evaluate LLMs in multi-turn interactions, Wang et al. [213] proposed MINT, which utilizes tools and natural language feedback.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [213] [213] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. arXiv preprint arXiv:2309.10691 (2023). - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    ToolBench project aims to empower development of large language models that effectively leverage capabilities of general-purpose tools.

    “the ToolBench project [191] aims to empower the development of large language models that effectively leverage the capabilities of general-purpose tools.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [191] [191] ToolBench. 2023. Open-source tools learning benchmarks. https://github.com/sambanova/toolbench. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.2 Human Evaluation · p. 28

    Bubeck et al. found that GPT-4 performs close to or even exceeds human performance on multiple tasks.

    “Bubeck et al. [15] did a series of human-crafted tests using GPT-4 and they found that GPT-4 performs close to or even exceeds human performance on multiple tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] [15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 5.1 Automatic Evaluation · p. 26

    Compared with human evaluation, automatic evaluation does not require intensive human participation, saves time, reduces impact of human subjective factors, and makes evaluation process more standardized.

    “Compared with human evaluation, automatic evaluation does not require intensive human participation, which not only saves time, but also reduces the impact of human subjective factors and makes the evaluation process more standardized.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    SEED-Bench evaluation covers 12 different aspects, including models' proficiency in understanding patterns within images and videos.

    “the evaluation covers 12 different aspects, including the models' proficiency in understanding patterns within images and videos.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [107] [107] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    LVLM-eHub undertakes an exhaustive evaluation of LVLMs using an online competitive platform and quantitative capacity assessments.

    “LVLM-eHub [231] undertakes an exhaustive evaluation of LVLMs using an online competitive platform and quantitative capacity assessments.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [231] [231] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. arXiv:2306.09265 [cs.CV] - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    MMICL enhances visual language models for multimodal inputs and excels in tasks such as MME and MMBench.

    “MMICL [255] enhances visual language models for multimodal inputs and excels in tasks such as MME and MMBench.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [255] [255] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2023. MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning. arXiv preprint arXiv:2309.07915 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § 5.1 Automatic Evaluation · p. 26

    Automated evaluation is a common, and perhaps the most popular, evaluation method that typically uses standard metrics and evaluation tools.

    “Automated evaluation is a common, and perhaps the most popular, evaluation method that typically uses standard metrics and evaluation tools to evaluate model performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    SEED-Bench consists of 19,000 multiple-choice questions annotated by human assessors.

    “This benchmark consists of 19,000 multiple-choice questions that have been annotated by human assessors.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [107] [107] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    LAMM extends its research to encompass multimodal point clouds.

    “LAMM [235] extends its research to encompass multimodal point clouds.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [235] [235] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. 2023. LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark. arXiv preprint arXiv:2306.06687 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 5.2 Human Evaluation · p. 28

    Human evaluations can have high variance and instability, which could be due to cultural and individual differences.

    “even human evaluations can have high variance and instability, which could be due to cultural and individual differences [155].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [155] [155] Kaiping Peng, Richard E Nisbett, and Nancy YC Wong. 1997. Validity problems comparing values across cultures and possible solutions. Psychological methods 2, 4 (1997), 329. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.2 Benchmarks for Specific Downstream Tasks · p. 25

    API-Bank comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.

    “It comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [109] [109] Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Benchmark for Tool-Augmented LLMs. arXiv:2304.08244 [cs.CL] - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.3 Benchmarks for Multi-modal task · p. 26

    SEED-Bench was introduced to comprehensively assess generative and understanding capabilities of MLLMs.

    “To comprehensively assess the generative and understanding capabilities of Multi-modal Large Language Models (MLLMs), Li et al. [107] introduced a novel benchmark named SEED-Bench.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [107] [107] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 6.1.1 What can LLMs do well? · p. 30

    LLMs exhibit robust contextual comprehension, enabling them to generate coherent responses that align with the given input.

    “LLMsexhibit robust contextual comprehension, enabling them to generate coherent responses that align with the given input [187].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [187] [187] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022). - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 6.1.2 When can LLMs fail? · p. 30

    LLMs have limited abilities on abstract reasoning and are prone to confusion or errors in complex contexts.

    “LLMs have limited abilities on abstract reasoning [56], and are prone to confusion or errors in complex contexts [148].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [56] [56] Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2023. Large Language Models Are Not Abstract Reasoners. arXiv preprint arXiv:2305.19555 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [148] [148] Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. ThoughtSource: A central hub for large language model reasoning data. arXiv preprint arXiv:2301.11596 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 6.1.2 When can LLMs fail? · p. 30

    LLMs exhibit restricted proficiency in discerning semantic similarity between events and demonstrate substandard performance in evaluating fundamental phrases.

    “LLMs exhibit restricted proficiency in discerning semantic similarity between events [184] and demonstrate substandard performance in evaluating fundamental phrases [166].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [184] [184] Zhengwei Tao, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Yanlin Feng, Jia Li, and Wenpeng Hu. 2023. EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models. arXiv preprint arXiv:2305.15268 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [166] [166] Nicholas Riccardi and Rutvik H Desai. 2023. The Two Word Test: A Semantic Benchmark for Large Language Models. arXiv preprint arXiv:2306.04610 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 6.1.2 When can LLMs fail? · p. 30

    Within NLI, LLMs exhibit subpar performance and encounter challenges in accurately representing human disagreements.

    “Within the realm of NLI, LLMs exhibit subpar performance and encounter challenges in accurately representing human disagreements [105].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [105] [105] Noah Lee, Na Min An, and James Thorne. 2023. Can Large Language Models Infer and Disagree Like Humans? arXiv preprint arXiv:2305.13788 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Premisecitation
    § 6.1.1 What can LLMs do well? · p. 30

    LLMs demonstrate robust arithmetic reasoning capabilities and excel in logical reasoning.

    “LLMsdemonstrate robust arithmetic reasoning capabilities [159] and excel in logical reasoning [124].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [124] [124] Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4. arXiv:2304.03439 [cs.CL] - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 7.7 Beyond Evaluation: LLMs Enhancement · p. 33

    PromptBench provides robustness evaluation results on adversarial prompts and a comprehensive analysis through attention visualization, elucidating how adversarial texts can result in erroneous responses.

    “PromptBench [264] provides not only robustness evaluation results on adversarial prompts but also a comprehensive analysis through attention visualization, elucidating how adversarial texts can result in erroneous responses.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [264] [264] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arXiv preprint arXiv:2306.04528 (2023). - source

  • May combine two claims.

    Premisecitation
    § 7.7 Beyond Evaluation: LLMs Enhancement · p. 33

    Wang et al. explored the performance of large vision-language models on imbalanced (long-tailed) tasks, demonstrating the limitation of current large models.

    “Wang et al. [215] first explored the performance of large vision-language models on imbalanced (long-tailed) tasks, which demonstrates the limitation of current large models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [215] [215] Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2023. Exploring Vision-Language Models for Imbalanced Learning. arXiv preprint arXiv:2304.01457 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 7.7 Beyond Evaluation: LLMs Enhancement · p. 33

    Wang et al. explored different methodologies to enhance performance on imbalanced (long-tailed) tasks.

    “Then, they explored different methodologies to enhance the performance on these tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [215] [215] Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2023. Exploring Vision-Language Models for Imbalanced Learning. arXiv preprint arXiv:2304.01457 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 6.2 Benchmark and Evaluation Protocol · p. 31

    A shift from a unified to a challenging setting in evaluating machine learning models is occurring.

    “Third, a shift from a unified to a challenging setting in evaluating machine learning models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 6.1.2 When can LLMs fail? · p. 31

    LLMs have limitations in incorporating real-time or dynamic information, making them less suitable for tasks requiring up-to-date knowledge or rapid adaptation.

    “LLMs have limitations in incorporating real-time or dynamic information [127], making them less suitable for tasks that require up-to-date knowledge or rapid adaptation to changing contexts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [127] [127] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.2 Benchmark and Evaluation Protocol · p. 31

    AdaFilter may not be entirely fair as it relies on adversarial examples.

    “However, it is worth noting that AdaFilter may not be entirely fair as it relies on adversarial examples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 7.4 Dynamic and Evolving Evaluation · p. 32

    Static and public benchmarks are likely to be memorized by LLMs, resulting in potential training data contamination.

    “On the other hand, as LLMs grow increasingly powerful with larger model sizes and training set sizes, static and public benchmarks are likely to be memorized by LLMs, resulting in potential training data contamination.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLMs grow increasingly powerful with larger model sizes and training set sizes.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 7.6 Unified Evaluation that Supports All LLMs Tasks · p. 32

    PandaLM is an evaluation system that assists LLMs fine-tuning by providing an open-source evaluation model, which can automatically assess the performance of fine-tuning.

    “PandaLM [216] is an evaluation system that assists LLMs fine-tuning by providing an open-source evaluation model, which can automatically assess the performance of fine-tuning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [216] [216] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. arXiv preprint arXiv:2306.05087 (2023). - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 6.2 Benchmark and Evaluation Protocol · p. 31

    A move from static to crowd-sourcing test sets is becoming more common.

    “Second, a move from static to crowd-sourcing test sets is becoming more common.”
    Quote in the paperNot a substantial claim
169 of 169

  • The paper relies on this without support.

    LLMs possess the capabilities to solve diverse tasks, contrasting with prior models confined to solving specific tasks.

    “LLMs possess the capabilities to solve diverse tasks, contrasting with prior models confined to solving specific tasks.”

  • The paper relies on this without support.

    LLMs are increasingly used by individuals with critical information needs, such as students or patients.

    “Due to its great performance in handling different applications such as general natural language tasks and domain-specific ones, LLMs are increasingly used by individuals with critical information needs, such as students or patients.”

  • The paper relies on this without support.

    LLMs are primarily evaluated on English data, leading to a potential oversight of evaluating their multilingual performance.

    “However, due to the relatively recent emergence of this technology, LLMs are primarily evaluated on English data, leading to a potential oversight of evaluating their multilingual performance.”

  • The paper relies on this without support.

    There is a research gap in assessing the effectiveness of utilizing tools for LLMs.

    “In addition to existing evaluation benchmarks, there is a research gap in assessing the effectiveness of utilizing tools for LLMs.”

  • The authors state this as an assumption.

    LLMs are expected to demonstrate AGI abilities.

    “As we expect LLMs to demonstrate AGI abilities, a comprehensive understanding of the differences between human and AGI capacities becomes crucial in the creation of AGI benchmarks.”

  • The paper relies on this without support.

    Current LLMs are not robust to input variations such as different grammars and expressions.

    “For instance, the same prompts but with different grammars and expressions could lead ChatGPT and other LLMs to generate diverse results, indicating that current LLMs are not robust to the inputs.”

  • The paper relies on this without support.

    The concept and definition of robustness are constantly evolving.

    “Concurrently, the concept and definition of robustness are constantly evolving.”

  • The paper relies on this without support.

    There is a scarcity of research in the domain of trustworthy evaluation.

    “There is a scarcity of research in this domain”

  • The paper relies on this without support.

    LLMs grow increasingly powerful with larger model sizes and training set sizes.

    “as LLMs grow increasingly powerful with larger model sizes and training set sizes”

  • The paper relies on this without support.

    LLMs develop at a rapid rate.

    “given their rapid rate of development”

  • The paper relies on this without support.

    Existing evaluation protocols for most AI tasks rely on static and public benchmarks.

    “Existing evaluation protocols for most AI tasks rely on static and public benchmarks, i.e., the evaluation datasets and protocols are often publicly available.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Context engineering adaptability is especially valuable in low-resource scenarios, enabling effective utilization across various prompt engineering techniques including zero-shot approaches, few-shot examples, and role context without requiring domain-specific fine-tuning.

    “This adaptability proves especially valuable in low-resource scenarios, enabling effective utilization across various prompt engineering techniques including zero-shot approaches, few-shot examples, and role context without requiring domain-specific fine-tuning [932, 129, 1083].”

    From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [129]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Comprehensive surveys of LLM evaluation emphasize the need to move beyond task performance toward a systematic discipline of measurement.

    “Comprehensive surveys of LLM evaluation emphasize the need to move beyond task performance toward a systematic discipline of measurement [22].”

    From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [22]

  • LLMs have demonstrated remarkable proficiency in processing semantic and contextual information.

    “These language models have demonstrated remarkable proficiency in processing semantic and contextual information [14], a key aspect of their effectiveness as a tool for learning.”

    From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [14]

  • The rapid development of LLMs has resulted in a significant lag in the development of evaluation methods/protocols.

    “The rapid development of large language models (LLMs) has resulted in a significant lag in the development of evaluation methods/protocols (Chang et al., 2024).”

    From Benchmarking Benchmark Leakage in Large Language Models (2024) · cites this paper as Chang et al., 2024

  • Defining a ground truth or evaluation dataset is an essential step in LLM performance evaluation.

    “Many studies describe this as defining a ground truth or evaluation dataset, and it is an essential step in LLM performance evaluation.166,167”

    From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [166]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • GAOKAO-Bench citation points to a different paper

    Minor

    The text cites reference [245] for GAOKAO-Bench, but [245] is listed as a paper on tool-augmented math reasoning, identical to reference [244].

  • Figure date range excludes a month it says it includes

    Minor

    A range ending in June 2023 cannot include July 2023.

  • Publication date predates research discussed in the paper

    Minor

    The paper gives an August 2018 publication date but discusses a study it identifies as being from 2023.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901.
  2. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321–1330.
  3. Ron Kohavi et al. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In IJCAI, Vol. 14. Montreal, Canada, 1137–1145.
  4. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013
  5. Microsoft. 2023. Bing chat. https://www.bing.com/new (2023).
  6. Andrew M. Olney. 2023. Generating multiple choice questions from a textbook: LLMs match human performance on most metrics. In AIED Workshops.
  7. Yujia Qin Shihao Liang Yining Ye Kunlun Zhu Lan Yan Yaxi Lu Yankai Lin Xin Cong Xiangru Tang Bill Qian Sihan Zhao Runchu Tian Ruobing Xie Jie Zhou Mark Gerstein Dahai Li Zhiyuan Liu and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. (2023). arxiv:cs.AI/2307.16789
  8. Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. In ICLR Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models.
  9. Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. PMLR, 325–333.

Cited by

Sorted by citations

Related

Authors
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X.
Affiliations
  • Jilin University (CN)
  • Microsoft Research Asia (China) (CN)
  • Westlake University (CN)
  • Chinese Academy of Sciences (CN)
  • Carnegie Mellon University (US)
  • Peking University (CN)
  • University of Illinois Chicago (US)
  • Hong Kong University of Science and Technology (HK)
ORCID
Venue
ACM Transactions on Intelligent Systems and Technology
Publisher
Association for Computing Machinery (ACM)
Published
2024-03-29
Volume
15
Issue
3
Pages
1-45
ISSN
  • 2157-6904 (ISSN-L)
  • 2157-6912
DOI
10.1145/3641289
OpenAlex
W4391136507
Cited by
2,842
References
293
Citations by year
2026: 819 · 2025: 1432 · 2024: 556 · 2023: 20
FWCI
439.94
Type
journal article
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Transactions on Intelligent Systems and Technology (publishedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
Funders
NSF (III-2106758)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, LLM evaluation, evaluation benchmark, assessment tasks, reasoning, medical applications, ethics, AI agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review