A Survey on Evaluation of Large Language Models

In the paper

“ChatGPT's sentiment analysis prediction performance is superior to traditional sentiment analysis methods [129] and comes close to that of GPT-3.5 [159].”

What the sources say

• Supported

  The results discussion following Figure 5 reports that GPT-4’s score retains predictive power for subsequent stock-price drift when included alongside RavenPack sentiment scores, while RavenPack’s coefficient becomes insignificant. The authors conclude that GPT-4 subsumes traditional sentiment analysis methods. The GPT-3.5 comparison has a separate citation in the citing sentence and is not attributed to this source here. (LLM, conf=0.92)

  cites [129] [129] Alejandro Lopez-Lira and Yuehua Tang. 2023. Can chatgpt forecast stock price movements? Return predictability and large language models. arXiv preprint arXiv:2304.07619 (2023). -
• Disputed

  Only the comparison with GPT-3.5 is attributed to this source by citation [159]. Section 4.2.8 says ChatGPT performed “much better” than GPT-3.5 overall on SST2; it says performance came close only for positively labeled samples. The claim applies that narrower finding to overall sentiment-analysis performance. (LLM, conf=0.84)

  Checked by a reasoning model — may be wrong.

  cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning [6].”

What the sources say

• Disputed

  The Commonsense Reasoning section (F.3) says ChatGPT performs quite well on three commonsense datasets, contrary to the claim that it generally performs poorly on those tasks. The Introduction does support the comparison: it says ChatGPT is relatively better at commonsense reasoning than non-textual semantic reasoning. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Furthermore, the quality of ChatGPT has shown improvement compared to GPT-4.”

What the sources say

• Disputed

  Section 3.8, “Comparison with GPT-4,” reports that GPT-4 significantly improved the quality of translated radiology reports compared with ChatGPT, not the reverse. The Discussions section likewise describes GPT-4 as a significant improvement over ChatGPT. Page numbers are not provided in the supplied passages. (LLM, conf=0.95)

  Checked by a reasoning model — may be wrong.

  cites [131] [131] Qing Lyu, Josh Tan, Mike E Zapadka, Janardhana Ponnatapuram, Chuang Niu, Ge Wang, and Christopher T Whitlow. 2023. Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential. arXiv preprint arXiv:2303.09038 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“The results show that the ChatGPT group spent less time on all tasks and the difference between these two groups is not significant.”

What the sources say

• Disputed

  Section 4.2 (Search Efficiency) supports the claim that the ChatGPT group spent less time on each task. It directly contradicts the claim that the time differences were not significant: the paper says they were statistically significant at the 1% level across all three tasks. (LLM, conf=0.95)

  Checked by a reasoning model — may be wrong.

  cites [232] [232] Ruiyun Xu, Yue Feng, and Hailiang Chen. 2023. ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience. arXiv preprint arXiv:2307.01135 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“GAOKAO-Bench [245] provides a comprehensive evaluation benchmark for gauging the proficiency of large language models in intricate and context-specific tasks, utilizing questions sourced from the Chinese Gaokao examination.”

What the sources say

• Disputed

  The Introduction identifies the benchmark developed in this paper as CARP, a Chinese dataset of computation-intensive algebra problems, not GAOKAO-Bench. Section 4.1 lists the additional datasets evaluated and does not identify GAOKAO-Bench among them. The source therefore contradicts the citation’s identification of it as the GAOKAO-Bench benchmark. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  cites [245] [245] Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2023. Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning. arXiv preprint arXiv:2306.02408 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“It comprises a comprehensive Tool-Augmented LLM workflow, encompassing 53 commonly used API tools and 264 annotated dialogues, encompassing a total of 568 API calls.”

What the sources say

• Disputed

  The Introduction reports 73 commonly used APIs, 314 manually annotated tool-use dialogues, and 753 API calls—not the claimed 53, 264, and 568. Section 3.2 independently confirms the 314 dialogues and 753 calls. The paper does describe API-Bank as a benchmark for tool-augmented LLMs, but its stated figures contradict every number in the claim. (LLM, conf=0.92)

  Checked by a reasoning model — may be wrong.

  cites [109] [109] Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Benchmark for Tool-Augmented LLMs. arXiv:2304.08244 [cs.CL] - sourceExperimental — accuracy not yet measured

In the paper

“LLMsdemonstrate robust arithmetic reasoning capabilities [159] and excel in logical reasoning [124].”

What the sources say

• Disputed

  The citation to this paper is attached to the logical-reasoning claim, not the arithmetic claim. In the Key takeaways (page unavailable), the paper says ChatGPT’s superior reasoning is substantiated in arithmetic tasks but that it often underperforms GPT-3.5 in logical reasoning tasks. Section 4.2.2 likewise reports weaker performance in many cases, rather than the claimed excellence in logical reasoning. (LLM, conf=0.82)

  Checked by a reasoning model — may be wrong.

  cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - sourceExperimental — accuracy not yet measured
• Partly supported

  The citation to this paper is attached to the logical-reasoning clause, not the arithmetic-reasoning clause. The Conclusion reports that ChatGPT and GPT-4 outperform a traditional fine-tuning method on most logical-reasoning benchmarks, but also says they struggle with new and out-of-distribution data and do not perform especially well on logical-reasoning NLI tasks. Thus, 'excel' overstates the paper's qualified finding. (LLM, conf=0.88)

  cites [124] [124] Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4. arXiv:2304.03439 [cs.CL] - sourceExperimental — accuracy not yet measured

In the paper

“Recently, large language models (LLMs) have incited substantial interest across both academic and industrial domains [11, 219, 257].”

• Also stated in: § A Survey on Evaluation of Large Language Models · p. 1 · plain

  “Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications.”

What the sources say

• Supported

  Section 2.1 (p. 22) says NLP has become largely centered on using and understanding foundation models. The discussion on p. 12 also describes strong economic incentives to advance these models and calls for collaboration between academia and industry. Together, these passages support the broad claim of substantial interest in language models across both domains. (LLM, conf=0.88)

  cites [11] [11] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). - source
• Supported

  The Introduction (p. 1) describes the impact of language models on NLP research. Section 5.5 (p. 10) describes applications beyond that research community and deployment in products and commercial services, supporting the broad statement that LLMs have attracted interest in both academic and industrial domains. (LLM, conf=0.86)

  cites [219] [219] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). - source
• Supported

  The Abstract (p. 1) says LLMs are reshaping both research and practical applications. The Introduction (p. 2) says ChatGPT sparked global interest in generative AI. Together, these passages support the broad claim of substantial academic and industrial interest in LLMs. (LLM, conf=0.88)

  cites [257] [257] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023). - source

In the paper

“Furthermore, GPT-3.5 and Bard cannot distinguish between meaningful and nonsense phrases, consistently classifying highly nonsense phrases as meaningful.”

What the sources say

• Partly supported

  The bTWT results report that GPT-3.5-turbo judged most pairs as making sense, including many nonsense pairs. The paper tests Gemini-1.0-Pro-001, not Bard, and says Gemini could distinguish sensible from nonsense phrases with modest discrimination, although it sometimes mislabeled nonsense phrases as meaningful. Thus the claim overstates the result and attributes it to a model the paper does not identify as tested. (LLM, conf=0.88)

  cites [166] [166] Nicholas Riccardi and Rutvik H Desai. 2023. The Two Word Test: A Semantic Benchmark for Large Language Models. arXiv preprint arXiv:2306.04610 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“GPT-4 shows significant improvements, but its performance is still significantly lower than that of humans.”

What the sources say

• Partly supported

  In the bTWT section (page unavailable), GPT-4-turbo shows modest discrimination compared with GPT-3.5-turbo’s poor discrimination, and Table 3 reports GPT-4-turbo’s performance as significantly different from humans (p < 0.001). The paper does not establish a statistically significant improvement for GPT-4; its explicit statement about a significant step forward refers to Claude-3-Opus. (LLM, conf=0.84)

  cites [166] [166] Nicholas Riccardi and Rutvik H Desai. 2023. The Two Word Test: A Semantic Benchmark for Large Language Models. arXiv preprint arXiv:2306.04610 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“They also found that ChatGPT excels in handling factual input that could be attributed to its RLHF training process in favoring human feedback.”

What the sources say

• Supported

  Section 4.2.3 (Natural Language Inference; page not provided) says ChatGPT is better at handling factual input and suggests this might be related to the preference for human feedback in its RLHF training. The citing sentence preserves that uncertainty with “could be attributed.” (LLM, conf=0.93)

  cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - source

In the paper

“Yang and Menczer [233] found that ChatGPT can produce credibility ratings for a wide range of news outlets, and these ratings have a moderate correlation with those from human experts.”

What the sources say

• Supported

  Section 4.1 reports testing LLM credibility ratings across 7,523 news sources. Section 4.2 reports that GPT-3.5’s ratings had a Spearman correlation of 0.50 with human expert ratings, supporting the described moderate correlation. (LLM, conf=0.88)

  cites [233] [233] Kai-Cheng Yang and Filippo Menczer. 2023. Large language models can rate news outlet credibility. arXiv preprint arXiv:2304.00228 (2023). - source

In the paper

“Specifically, ChatGPT demonstrates comparable zero-shot performance to the text-davinci-002 [6], but performs worse than GPT-3.5 [159].”

What the sources say

• Supported

  Section C.1 (Summarization) states that ChatGPT achieved similar zero-shot summarization performance to text-davinci-002. The worse-than-GPT-3.5 comparison has a separate citation and is not attributed to this source. (LLM, conf=0.93)

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source
• Supported

  The Key takeaways and Section 4.2.6 state that ChatGPT performs worse than GPT-3.5 on summarization; Table 9 reports lower ROUGE scores for ChatGPT on SAMSum. The text-davinci-002 comparison is attached to a different citation, so it is not assessed here. (LLM, conf=0.93)

  cites [159] [159] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023). - source

In the paper

“Meanwhile, ChatGPT also lacks spatial reasoning ability, but exhibits better temporal reasoning.”

What the sources say

• Supported

  The Introduction states that ChatGPT lacks spatial reasoning while showing better temporal reasoning. Section 3 reports weak performance on spatial tasks and 86.67% accuracy on a temporal-reasoning task, supporting that summary. (LLM, conf=0.95)

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

In the paper

“For multi-step reasoning, Fu et al. [47] showed PaLM and Claude2 are the only two model families that achieve similar performance (but still worse than the GPT model family).”

What the sources say

• Supported

  The Introduction states that, as of May 2023, PaLM and Claude 2 were the only two model families comparable to, but slightly worse than, the GPT family. The claim closely paraphrases that finding; its time frame is May 2023. (LLM, conf=0.94)

  cites [47] [47] Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance. arXiv preprint arXiv:2305.17306 (2023). - source

In the paper

“On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”

What the sources say

• Partly supported

  Section 3, “Reasoning Evaluations of ChatGPT” (page unavailable), reports poor performance on some reasoning tasks, especially induction. It does not establish that ChatGPT is mostly worse than GPT-3.5 on symbolic reasoning tasks or suggest that uncertain responses cause the poor performance. The passages describing uncertain responses concern language identification, not reasoning. (LLM, conf=0.78)

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Moreover, LLaMA-65B is the most robust open-source LLMs to date, which performs closely to code-davinci-002.”

What the sources say

• Supported

  The Introduction calls LLaMA 65B “the strongest open LLM to date” and says it performs closely to code-davinci-002. The Conclusion repeats the performance comparison. In this context, “most robust” is a reasonable paraphrase of “strongest.” (LLM, conf=0.92)

  cites [47] [47] Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023. Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance. arXiv preprint arXiv:2305.17306 (2023). - source

In the paper

“Liang et al. [114] found that TNLG v2 (530B) [179] achieved the highest score in both scenarios, followed by OPT (175B) [247] in second place.”

What the sources say

• Supported

  The Summarization section reports that TNLG v2 (530B) had the highest ROUGE-2 score on both CNN/DailyMail and XSUM. It identifies OPT (175B) as second-place for XSUM; it does not specify OPT’s rank on CNN/DailyMail. (LLM, conf=0.92)

  cites [114] [114] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022). - source

In the paper

“In professional domain reasoning tasks, zero-shot InstructGPT and Codex are capable of complex medical reasoning tasks, but still need to be further improved [117].”

What the sources say

• Supported

  The Conclusion (p. 10) says the models can often handle complex medical questions and perform nontrivial reasoning, while InstructGPT and Codex still make mistakes. The Discussion (p. 10) identifies limitations in Codex’s zero-shot performance, and the paper reports strong zero-shot results for InstructGPT and medical-question performance for zero-shot Codex. (LLM, conf=0.88)

  cites [117] [117] Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143 (2022). - source

In the paper

“Research by Bang et al. [6] underscores that fully fine-tuned models tailored for specific tasks surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.”

What the sources say

• Supported

  The Multitask Generalization of ChatGPT section states that, in most tasks—especially task-oriented and knowledge-grounded dialogue—task-specific fully fine-tuned models outperform ChatGPT. This directly supports the cited comparison, with the paper's 'in most tasks' qualification. (LLM, conf=0.93)

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

In the paper

“Finally, while the performance of ChatGPT is acceptable on causal and analogical reasoning, it performs poorly on multi-hop reasoning ability, which is similar to the weakness of other LLMs on complex reasoning [148].”

What the sources say

• Partly supported

  The paper’s background discussion says LLMs often fail at complex reasoning, including multi-hop question answering. It does not assess ChatGPT’s performance or compare its causal, analogical, and multi-hop reasoning abilities, so the ChatGPT-specific part is unsupported. (LLM, conf=0.86)

  cites [148] [148] Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. ThoughtSource: A central hub for large language model reasoning data. arXiv preprint arXiv:2301.11596 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Zheng et al. [259] have curated a comprehensive LLMs conversation dataset, LMSYS-Chat-1M, encompassing up to one million samples.”

What the sources say

• Supported

  The Conclusion states that the authors introduce LMSYS-Chat-1M as a dataset containing one million LLM conversations. The Basic Statistics section also reports 1,000,000 conversations, supporting the citing sentence’s description of its size. (LLM, conf=0.95)

  cites [259] [259] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. arXiv preprint arXiv:2309.11998 (2023). - source

In the paper

“The findings from Bang et al. [6] indicated that ChatGPT performs X → Eng translation well, but it still lacks the ability to perform Eng → X translation.”

What the sources say

• Supported

  Section C.2, “Machine Translation,” explicitly concludes that ChatGPT performs XXX→English translation well but lacks the ability to perform English→XXX translation. The citing sentence closely paraphrases that conclusion, which is based on the paper’s evaluated language pairs. (LLM, conf=0.95)

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - source

In the paper

“Fine-tuned models, such as Vícuna and ChatGPT, exhibit exceptional performance with near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin [5, 6].”

What the sources say

• Supported

  Table 3 (section 4, Experiments) reports full-mark scores of 96.5% for Vicuna-13B and 99.3% for ChatGPT, versus at most 62.9% for the listed models without supervised fine-tuning. Section 5 describes the fine-tuned models' scores as near-perfect. (LLM, conf=0.94)

  cites [5] [5] Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2023. Benchmarking Foundation Models with Language-Model-as-an-Examiner. arXiv preprint arXiv:2306.04181 (2023). - source
• Partly supported

  The Introduction reports that ChatGPT outperformed previous zero-shot models on 9 of 13 NLP datasets, supporting a narrower performance comparison. The Multitask Generalization section says fully fine-tuned models outperformed ChatGPT on most tasks. The paper does not establish near-perfect scores, a significant margin across models lacking supervised fine-tuning, or the claim about Vícuna. (LLM, conf=0.86)

  cites [6] [6] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Chen et al. [20] revealed that ChatGPT excels in assessing text quality from multiple angles, even in the absence of reference texts, surpassing the performance of most existing automated metrics.”

What the sources say

• Supported

  The Introduction (§1) says ChatGPT can assess text quality without references and outperforms commonly used metrics. The Meta Evaluation Strategy section adds that its Explicit Score correlates with human scores better than most commonly used automated metrics; §2.1 describes scoring specific aspects or overall quality. (LLM, conf=0.94)

  cites [20] [20] Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723 (2023). - source

In the paper

“demonstrating a propensity for progressive views and an ENFJ personality type.”

What the sources say

• Partly supported

  The Political Compass section reports that ChatGPT exhibits a left-libertarian ideology, supporting the progressive-views portion of the claim for ChatGPT. The paper does not report an ENFJ personality type, and its finding about ChatGPT alone does not establish the same propensity across LLMs generally (searched: ENFJ, Myers-Briggs, personality questionnaires). (LLM, conf=0.88)

  cites [65] [65] Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768 (2023). - sourceExperimental — accuracy not yet measured
• Partly supported

  The paper’s Political Biases results report that ChatGPT’s questionnaire answers indicated a bias toward progressive views. Its Personality Traits results say ChatGPT averaged an ENFJ result on the MBTI test, although four of ten runs yielded INFJ. The study tested ChatGPT, so it does not establish these tendencies for LLMs generally. (LLM, conf=0.85)

  cites [167] [167] Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, and Markus Pauly. 2023. The Self-Perception and Political Biases of ChatGPT. arXiv preprint arXiv:2304.07333 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“[256] proposes a Chinese conversational bias evaluation dataset CHBias, discovers bias risks in pre-trained models, and explores debiasing methods.”

What the sources say

• Supported

  The Introduction says the authors introduced CHBias, used it to evaluate two Chinese pretrained dialogue models, and found that their responses exhibited social biases. It also says they applied debiasing methods. The claim is supported in the context of those Chinese conversational models, not pretrained models generally. (LLM, conf=0.93)

  cites [256] [256] Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, and Mykola Pechenizkiy. 2023. CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models. arXiv:2305.11262 [cs.CL] - source

In the paper

“The study conducted by [69] reveals that existing LMs have potential in ethical judgment, but still need improvement.”

What the sources say

• Supported

  The Introduction says existing language models have “low but promising performance” on the ETHICS dataset and “have much to learn” about morally salient features. This supports the claim that they show potential in ethical judgment but need improvement. (LLM, conf=0.94)

  cites [69] [69] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275 (2020). - source

In the paper

“This dataset has become widely used for evaluating the factuality of LLMs [89, 146, 192, 220].”

What the sources say

• Partly supported

  Section 1.2 (p. 5) lists TruthfulQA among the paper’s evaluation tasks, and Section 1.3 (p. 7) identifies it as a benchmark for truthfulness, which the paper relates to factual accuracy. The paper establishes its use for this purpose but does not establish that it had become widely used. (LLM, conf=0.78)

  cites [89] [89] Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. ArXiv ab sourceExperimental — accuracy not yet measured
• Supported

  In 5 Limitations (p. 10), the report treats TruthfulQA as a public benchmark for testing whether models distinguish facts from incorrect statements. Figure 7 (p. 11) uses it to compare GPT-4 with other LLMs, supporting the broad characterization of its use for factuality evaluation. (LLM, conf=0.84)

  cites [146] [146] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] - source
• Supported

  Section 5.4 (pp. 9–10) describes TruthfulQA as a benchmark for measuring model truthfulness and uses it to evaluate LLaMA alongside reported GPT-3 results. That treatment supports the broad statement that it is used to evaluate LLM factuality. (LLM, conf=0.84)

  cites [192] [192] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). - source
• Partly supported

  Section 3 (p. 3) uses TruthfulQA to evaluate whether language models answer questions truthfully, and §5.4 (p. 9) discusses results for GPT-3 and Gopher. These passages support its use as an evaluation benchmark, but do not establish that it had become widely used. (LLM, conf=0.82)

  cites [220] [220] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res. 2022 (2022). - sourceExperimental — accuracy not yet measured

In the paper

“The results of the study indicated that while GPT-4 and BingChat can provide correct answers for more than 80% of the questions, there is still a remaining gap of over 15% to achieve complete accuracy.”

What the sources say

• Partly supported

  Section 5.1 says GPT-4 and BingChat performed well but did not achieve perfect accuracy. Table 4 reports human-evaluated accuracy of 78.8% and 79.9% on Natural Questions, and 90.2% and 89.6% on TriviaQA. Thus, neither model exceeded 80% on Natural Questions, and neither had a gap of over 15 percentage points on TriviaQA. (LLM, conf=0.90)

  cites [204] [204] Cunxiang Wang, Sirui Cheng, Zhikun Xu, Bowen Ding, Yidong Wang, and Yue Zhang. 2023. Evaluating open question answering evaluation. arXiv preprint arXiv:2305.12421 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“The study demonstrated that this approach surpasses traditional ranking methods by achieving an accuracy improvement of over 30%.”

What the sources say

• Supported

  The Abstract reports that the paper’s information-theory-based metrics surpassed previous ranking-based methods by over 35% in a synthetic experiment. This supports the claimed improvement of over 30%, with the qualification that the reported comparison was in a synthetic experiment. (LLM, conf=0.91)

  cites [156] [156] Pouya Pezeshkpour. 2023. Measuring and Modifying Factual Knowledge in Large Language Models. arXiv preprint arXiv:2306.06264 (2023). - source

In the paper

“they conducted a review of current factual consistency evaluation methods and highlighted the absence of a unified comparison framework and the limited reference value of related scores compared to binary labels.”

What the sources say

• Supported

  The Introduction says there was no agreed-upon meta-evaluation protocol and that TRUE surveys methods using a unified binary-label format. Section 2.3 says previously reported system-level correlation scores are not very informative for assessing example-level binary inconsistency detection. The limitation applies to those correlation scores, not to metric scores generally. (LLM, conf=0.88)

  cites [74] [74] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991 (2022). - source

In the paper

“They revealed that while GPT-4 often showcases improved trustworthiness over GPT-3.5 in standard evaluations, it is simultaneously more susceptible to attacks.”

What the sources say

• Partly supported

  The Abstract and Conclusion report that GPT-4 generally performs better than GPT-3.5 on standard evaluations but is easier to manipulate with jailbreaking or misleading prompts. The claim broadens that finding to attacks in general; the Introduction’s adversarial-robustness summary also reports attacks against which GPT-4 is more resistant than GPT-3.5. (LLM, conf=0.85)

  cites [201] [201] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. arXiv:2306.11698 [cs.CL] - sourceExperimental — accuracy not yet measured

In the paper

“Deshpande et al. [35] introduced role-playing into the model and observed an increase in generated toxicity up to 6x. Furthermore, such role-playing also caused biased toxicity towards specific entities.”

What the sources say

• Supported

  The Introduction states that assigning a persona can increase ChatGPT's toxicity up to sixfold and that its outputs target specific entities and groups. Section 4.4 further reports that toxicity varies across entities, supporting the claim about biased toxicity. (LLM, conf=0.94)

  cites [35] [35] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335 (2023). - source

In the paper

“They found that these models can avoid common human intuitions and cognitive errors, demonstrating super-rational performance.”

What the sources say

• Supported

  The Abstract states that LLMs with higher cognitive capabilities, particularly ChatGPT and GPT-4, avoid the cognitive errors associated with human-like intuition and perform in a “hyperrational manner.” This supports the citing sentence’s paraphrase of super-rational performance. (LLM, conf=0.94)

  cites [62] [62] Thilo Hagendorff and Sarah Fabi. 2023. Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models - and Disappeared in GPT-4. arXiv:2306.07622 [cs.CL] - source

In the paper

“The research discovered that fact evaluation methods founded on natural language inference and question generation answering exhibit superior performance and can complement each other.”

What the sources say

• Supported

  The Introduction reports that NLI and QG-QA approaches perform significantly better across a wide variety of tasks and datasets, and that combining them yields even better results. The Conclusions likewise describes the approaches as performing well and being highly complementary. (LLM, conf=0.94)

  cites [74] [74] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991 (2022). - source

In the paper

“finding that it still exhibits noxious content to some extend.”

What the sources say

• Supported

  The Toxicity results (p. 9) report a small but nonzero toxic fraction for ChatGPT (0.005). The Beyond Benchmarking case study (p. 10) also reports toxic responses under prompt injection, while noting that ChatGPT normally avoids generating toxic content. (LLM, conf=0.94)

  cites [266] [266] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867 (2023). - source

In the paper

“LLMs have been found to internalize, spread, and potentially magnify harmful information existing in the crawled training corpora, usually, toxic languages, like offensiveness, hate speech, and insults [53], as well as social biases like stereotypes towards people with a particular demographic identity ( e.g. , gender, race, religion, occupation, and ideology) [175].”

What the sources say

• Supported

  The Introduction and Choice of Pretraining Data (pages not supplied) report toxic content in web-text training corpora and link it to models’ toxic generations. The paper also finds that models generate toxic text from non-toxic prompts, supporting the possibility of magnifying harmful content. The separate claim about demographic stereotypes is attached to citation [175], not this source. (LLM, conf=0.90)

  cites [53] [53] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020 . 3356-3369. - source
• Supported

  The citation [175] is attached to the social-bias portion of the sentence, not the examples of toxic language. Section 4.1 describes biased web training data and associations learned by large language models; “Representational Impacts” describes generated text propagating stereotypes about demographic groups, and Section 4 says parts of the generation pipeline can exacerbate bias. The listed identities are illustrative examples, though the passages do not specifically discuss ideology. (LLM, conf=0.88)

  cites [175] [175] Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal Biases in Language Generation: Progress and Challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . 4275-4293. - source

In the paper

“Wu et al. [225] evaluated the use of GPT-4 on difficult high school competition problems and GPT-4 reached 60% accuracy on half of the categories. Intermediate algebra and precalculus can only be solved with a low accuracy rate of around 20%.”

What the sources say

• Partly supported

  The evaluation’s Table 1 reports approximately 60% accuracy in three of six categories for MathChat using GPT-4, not for GPT-4 under vanilla prompting; the latter is identified as showing GPT-4’s native capability and does not reach 60% in any category. The same passage explicitly says that Intermediate Algebra and Precalculus were solved at around 20% accuracy across the methods considered. The citing sentence omits the MathChat-method qualifier on its 60% result. (LLM, conf=0.88)

  cites [225] [225] Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. 2023. An Empirical Study on Challenging Math Problem Solving with GPT-4. arXiv preprint arXiv:2306.01337 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“LLMs have several problems, including incomplete sentences and words, meaningless sentences merge, and more serious errors such as inconsistent and hallucinated information [34].”

What the sources say

• Partly supported

  Section 6 reports merged sentences that leave sentences incomplete in summaries generated by abstractive models and LLMs; the Introduction also says their summaries contain inconsistent or hallucinated information. The copy does not establish incomplete words as an LLM error: the word-merging examples in Table 7 are attributed to LegLED and LegPegasus-IN, which the paper treats as pre-trained summarization models. (LLM, conf=0.82)

  cites [34] [34] Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2023. How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization? arXiv preprint arXiv:2306.01248 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Specifically, GPT-3 demonstrates nearly random performance, while GPT-3.5 shows improvement, and GPT-4 performs the best [3].”

What the sources say

• Supported

  The General trends section explicitly says GPT-3 performs near random on JEEBENCH, GPT-3.5 performs better, and GPT-4 is far superior to GPT-3.5. Table 2 shows the same ordering for mathematics questions specifically (0.107, 0.146, and 0.212, respectively, versus 0.105 for random guessing). No page number is supplied in our copy. (LLM, conf=0.94)

  cites [3] [3] Daman Arora, Himanshu Gaurav Singh, et al. 2023. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. arXiv preprint arXiv:2305.15074 (2023). - source

In the paper

“Despite the advancements made in the new models, it is important to note that the peak performance remains relatively low compared to that of experts and these models lack the capability to engage in mathematical research [15].”

What the sources say

• Supported

  The opening of “Mathematical abilities” states that GPT-4, despite surpassing previous LLMs in mathematics, is still far from expert level and lacks the capacity to conduct mathematical research. This directly supports the citing sentence’s point about peak model performance. (LLM, conf=0.91)

  cites [15] [15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). - source

In the paper

“On the other hand, LLMs exhibit competence in handling decimal numbers, negative numbers, and irrational numbers [241].”

What the sources say

• Supported

  The Grouped Results section explicitly says that LLMs have some ability to deal with decimal, negative, and irrational numbers. The citing sentence paraphrases that finding, though the paper characterizes the ability as limited. (LLM, conf=0.95)

  cites [241] [241] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015 (2023). - source

In the paper

“LLMs are capable of generating coherent and seemingly factual text. However, the information generated can include factual inaccuracies or statements ungrounded in reality, a phenomenon known as hallucination [163, 253].”

What the sources say

• Supported

  Section 1.2 defines hallucination as generated content that is not factual or accurate. It also says models can produce plausible-sounding text containing fictional or fabricated claims that do not align with reality. No page number is provided in our copy. (LLM, conf=0.95)

  cites [163] [163] Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. arXiv preprint arXiv:2309.05922 (2023). - source
• Supported

  Section 2.2 defines fact-conflicting hallucination as content inconsistent with established world knowledge or unverifiable against it. “Invisibility of errors” says LLM-generated false information often appears highly plausible, supporting the contrast between seemingly factual text and inaccuracies. (LLM, conf=0.93)

  cites [253] [253] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309.01219 [cs.CL] - source

In the paper

“For fundamental mathematical problems, most large language models (LLMs) demonstrate proficiency in addition and subtraction, and possess some capability in multiplication.”

What the sources say

• Supported

  The Grouped Results section directly states that most LLMs can do addition and subtraction and have some ability in multiplication. The claim is a faithful paraphrase of that finding. (LLM, conf=0.95)

  cites [241] [241] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015 (2023). - source

In the paper

“found that LLMs can generate meaningful responses regarding political ideology and significantly improve text-as-data methods in social science.”

What the sources say

• Partly supported

  The Discussion reports that the LLM’s pairwise responses yield sensible scales of lawmakers’ ideology, supporting the claim that it can generate meaningful responses on political ideology. The Abstract and Discussion describe a promising measurement approach, but they do not establish that LLMs significantly improve text-as-data methods across social science. (LLM, conf=0.84)

  cites [224] [224] Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing. 2023. Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting. arXiv preprint arXiv:2303.12057 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“When it comes to generation tasks, LLMs often produce explanations that surpass the quality of gold references provided by crowd workers.”

What the sources say

• Partly supported

  Section 7.2 reports that LLM generations were rated superior to human gold annotations over 38% of the time across five generation tasks. The Human Ranking Evaluation section says the best models approached parity, not substantial overall superiority. It identifies the references as human gold annotations but does not establish that crowd workers provided them; the authors, rather than crowd workers, performed the reported rankings. (LLM, conf=0.85)

  cites [269] [269] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:2305.03514 (2023). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 sourceExperimental — accuracy not yet measured

In the paper

“experimental results demonstrated that LRV-Instruction effectively alleviates illusions in LLMs.”

What the sources say

• Partly supported

  The paper’s ‘Why can't LMMs accurately follow human instructions?’ section reports that models fine-tuned on LRV-Instruction suffered much less hallucination. The tested models were large multimodal models (LMMs), however, not LLMs generally; that scope of the claim is unsupported. (LLM, conf=0.88)

  cites [123] [123] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning. arXiv:2306.14565 [cs.CV] - sourceExperimental — accuracy not yet measured

In the paper

“However, they face challenges when it comes to division, exponentiation, trigonometry functions, and logarithm functions.”

What the sources say

• Supported

  The Grouped Results section explicitly says that division, exponentiation, trigonometry functions, and logarithm functions are hard for most LLMs. The claim paraphrases that finding, though the paper qualifies it as applying to most LLMs rather than all of them. (LLM, conf=0.95)

  cites [241] [241] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015 (2023). - source

In the paper

“Nay et al. [139] indicated that LLMs, particularly when combined with prompting enhancements and the correct legal texts, could perform better but not yet at expert tax lawyer levels.”

What the sources say

• Supported

  The experimental results report that few-shot prompting improved GPT-4’s performance and that combining prompting enhancements with the most relevant legal text produced its best accuracy. The Implications section says even the best models underperform a professional tax lawyer. Page numbers are not available in our copy. (LLM, conf=0.94)

  cites [139] [139] John J Nay, David Karamardian, Sarah B Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H Choi, and Jungo Kasai. 2023. Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence. arXiv preprint arXiv:2306.07075 (2023). - source

In the paper

“Conversely, LLMs achieve the best performance on misinformation, stance, and emotion classification.”

What the sources say

• Supported

  Section 5.1 reports the best zero-shot score on misinformation detection (77.4 F1) and identifies stance (76.0 F1) and emotion (70.8 F1) among the tasks with the highest absolute performance. The claim accurately paraphrases these results for the classification tasks evaluated; page numbers are not provided in our copy. (LLM, conf=0.91)

  cites [269] [269] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:2305.03514 (2023). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 source

In the paper

“Specifically, ChatGPT has developed superior skills in dynamic programming, greedy algorithm, and search, surpassing highly capable college students, but it struggles in data structure, tree, and graph theory.”

What the sources say

• ?Could not verify

  The E X A M section (page unavailable) ranks ChatGPT among other models, and Additional information (page unavailable) discusses variability in its answers. Neither reports a comparison with highly capable college students or performance in the named computer-science topics. Searches for those topics and the student comparison found no passage addressing the claim. (LLM, conf=0.75)

  cites [265] [265] Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. 2023. Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective. arXiv preprint arXiv:2306.10512 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Chervenak et al. [21] highlighted that while ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts, its limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.”

What the sources say

• Supported

  The Abstract reports that ChatGPT produced responses to fertility-related clinical queries comparable to established sources. It also identifies unreliable source citation and the possibility of fabricated information as limitations on clinical use; the Results provide the comparison with CDC infertility FAQs. (LLM, conf=0.94)

  cites [21] [21] Joseph Chervenak, Harry Lieman, Miranda Blanco-Breindel, and Sangita Jindal. 2023. The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertility and Sterility (2023). - source

In the paper

“The results indicate an overall accuracy of 46.8% for GPT-3.5 and 76.4% for GPT-4, demonstrating a significant performance difference between the two models.”

What the sources say

• Supported

  The Results section reports 46.8% overall accuracy for GPT-3.5 and 76.4% for GPT-4 on 280 Korean general surgery board-exam questions. It also reports a statistically significant difference between the models (P < 0.001). (LLM, conf=0.95)

  cites [143] [143] Namkee Oh, Gyu-Seong Choi, and Woo Yong Lee. 2023. ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models. Annals of Surgical Treatment and Research 104, 5 (2023), 269. - source

In the paper

“In the field of medical assistance, LLMs demonstrate potential applications, including research on identifying gastrointestinal diseases [99], dementia diagnosis [217], accelerating the evaluation of COVID-19 literature [93], and their overall potential in healthcare [17].”

What the sources say

• Partly supported

  The Abstract and Discussion describe ChatGPT’s potential to identify research priorities and generate research questions in gastroenterology. They do not report using an LLM to identify gastrointestinal diseases, which is the application attributed to this source by citation [99]. The dementia, COVID-19, and broader healthcare examples have separate citations in the citing sentence. (LLM, conf=0.84)

  cites [99] [99] Adi Lahat, Eyal Shachar, Benjamin Avidan, Zina Shatz, Benjamin S Glicksberg, and Eyal Klang. 2023. Evaluating the use of large language model in identifying top research questions in gastroenterology. Scientific reports 13, 1 (2023), 4164. - sourceExperimental — accuracy not yet measured
• Supported

  Citation [217] is attached specifically to dementia diagnosis, not to the other medical applications in the sentence. The paper’s Abstract says GPT-4 shows potential for future advancements in dementia diagnosis, while noting that it does not currently outperform traditional AI tools. (LLM, conf=0.94)

  cites [217] [217] Zhuo Wang, Rongzhen Li, Bowen Dong, Jie Wang, Xiuxing Li, Ning Liu, Chenhui Mao, Wei Zhang, Liling Dong, Jing Gao, et al. 2023. Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today. arXiv preprint arXiv:2306.01499 (2023). - source
• Supported

  Citation [93] is attached specifically to accelerating the evaluation of COVID-19 literature, not to the other medical applications in the sentence. The Abstract describes covLLM as a potential tool to assist clinicians in evaluating coronavirus literature, and the Discussion says it could help scientists and clinicians rapidly incorporate findings. (LLM, conf=0.93)

  cites [93] [93] Yousuf A Khan, Clarisse Hokia, Jennifer Xu, and Ben Ehlert. 2023. covLLM: Large Language Models for COVID-19 Biomedical Literature. arXiv preprint arXiv:2306.04926 (2023). - source

In the paper

“In software engineering tasks, ChatGPT generally performs well and provides detailed responses, often surpassing both human expert output and SOTA output.”

What the sources say

• Supported

  The Abstract (page unavailable) says that ChatGPT performed credibly on many of the fifteen software engineering tasks and that its responses were detailed and often better than human expert or state-of-the-art output. The Conclusion (page unavailable) adds that performance varied by task, including some poor results; the claim’s “generally” and “often” are consistent with that qualification. (LLM, conf=0.94)

  cites [181] [181] Giriprasad Sridhara, Sourav Mazumdar, et al. 2023. ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks. arXiv preprint arXiv:2305.16837 (2023). - source

In the paper

“The results indicated that ChatGPT achieves varying accuracies across different datasets.”

What the sources say

• Supported

  The Results > Overall Performance passage reports different ChatGPT accuracies across the AMBOSS and NBME Step 1 and Step 2 question sets: 44%, 42%, 64.4%, and 57.8%, respectively. This directly supports the attributed claim that accuracy varied across datasets. (LLM, conf=0.94)

  cites [57] [57] Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does CHATGPT perform on the United States Medical Licensing Examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education 9, 1 (2023), e45312. - source

In the paper

“However, for certain tasks such as code vulnerability detection and information retrieval-based test prioritization, the current version of ChatGPT fails to provide accurate answers, rendering it unsuitable for these specific tasks [181].”

What the sources say

• Supported

  The Defect or Vulnerability Detection section reports that ChatGPT identified only four of ten vulnerabilities and concludes it could not be used as-is for that task. The Test Case Prioritization section says ChatGPT could not perform information retrieval-based prioritization and was perhaps not viable for it in its current state. The Conclusion clarifies that it provided no answer at all for that task, rather than an inaccurate one. (LLM, conf=0.91)

  cites [181] [181] Giriprasad Sridhara, Sourav Mazumdar, et al. 2023. ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks. arXiv preprint arXiv:2305.16837 (2023). - source

In the paper

“The findings demonstrate that ChatGPT effectively translates radiology reports into accessible language and provides general recommendations.”

What the sources say

• Supported

  The Discussion says ChatGPT translated radiology reports into clear, easily understood language. Section 3.3 reports that it provided relevant general suggestions for patients and healthcare providers, such as following up with doctors. The paper also notes that translations could sometimes omit or oversimplify information. (LLM, conf=0.92)

  cites [131] [131] Qing Lyu, Josh Tan, Mike E Zapadka, Janardhana Ponnatapuram, Chuang Niu, Ge Wang, and Christopher T Whitlow. 2023. Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential. arXiv preprint arXiv:2303.09038 (2023). - source

In the paper

“[3] showed that LLMs perform worse on physics problems than chemistry problems, probably because chemistry problems have lower inference complexity than physics problems in this setting.”

What the sources say

• Supported

  The General trends section says the performance boost is greater for Chemistry than for Physics and suggests this is probably because reasoning complexity is lowest in Chemistry among the JEEBENCH subjects. That supports the citing sentence’s hedged explanation in this benchmark setting. (LLM, conf=0.94)

  cites [3] [3] Daman Arora, Himanshu Gaurav Singh, et al. 2023. Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. arXiv preprint arXiv:2305.15074 (2023). - source

In the paper

“Chervenak et al. [21] highlighted that while ChatGPT can generate responses similar to existing sources in fertility-related clinical prompts, its limitations in reliably citing sources and potential for fabricating information restrict its clinical utility.”

What the sources say

• Supported

  The Abstract says ChatGPT produced fertility-related clinical responses comparable to established sources, but that unreliable source citation and the possibility of fabricated information may limit its clinical use. The Results also report that only one of 147 factual statements cited a source and that nine were incorrect. (LLM, conf=0.95)

  cites [21] [21] Joseph Chervenak, Harry Lieman, Miranda Blanco-Breindel, and Sangita Jindal. 2023. The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertility and Sterility (2023). - source

In the paper

“Moreover, Sharma et al. [173] found that answers generated by ChatGPT are more context-aware with better deductive reasoning abilities compared to Google search results.”

What the sources say

• Supported

  The Conclusion explicitly states that ChatGPT-generated answers were more context-oriented and demonstrated better deductive reasoning abilities than regular Google search results. The Discussion makes the same comparison. (LLM, conf=0.94)

  cites [173] [173] Prabin Sharma, Kisan Thapa, Prastab Dhakal, Mala Deep Upadhaya, Santosh Adhikari, and Salik Ram Khanal. 2023. Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education. arXiv preprint arXiv:2307.00112 (2023). - source

In the paper

“Moreover, it explored the capability of LLMs for plan generalization and found that their generalization capabilities appear to be limited.”

What the sources say

• Supported

  The Introduction explicitly says that LLMs’ generalization capabilities seem limited. Section 5.2 supports this assessment: neither evaluated model met the paper’s threshold for plan-length generalization. (LLM, conf=0.95)

  cites [150] [150] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia. 2023. Understanding the Capabilities of Large Language Models for Automated Planning. arXiv preprint arXiv:2305.16151 (2023). - source

In the paper

“ChatGPT generated relatively accurate information for various medical queries, including genetics [39], radiation oncology physics [73], biomedicine [81], and many other medical disciplines [63, 87, 169], demonstrating its effectiveness in the field of medical queries to a certain extent.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [39] [39] Dat Duong and Benjamin D Solomon. 2023. Analysis of large-language model versus human performance for genetics questions. European Journal of Human Genetics (2023), 1-3. -
• Supported

  The citation to this paper supports the radiation oncology physics example, not the other disciplines listed alongside it. Section 4.1 reports that ChatGPT (GPT-4) answered 67% of the tested questions correctly in each trial, and the Discussion describes its performance on this highly specialized topic as extraordinary. (LLM, conf=0.89)

  cites [73] [73] Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T Sio, Lisa A McGee, Jonathan B Ashman, Xiang Li, Tianming Liu, Jiajian Shen, et al. 2023. Evaluating large language models on a highly-specialized topic, radiation oncology physics. arXiv preprint arXiv:2304.01938 (2023). - source
• Supported

  The Question Answering Evaluation reports 51.60% accuracy for zero-shot ChatGPT on PubMedQA, providing support for the citation’s qualified claim about biomedical queries. The paper also says ChatGPT may be useful in low-resource biomedical tasks, while noting that its question-answering accuracy was much lower than BioGPT’s. (LLM, conf=0.82)

  cites [81] [81] Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023. Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers. arXiv preprint arXiv:2306.04504 (2023). - source

In the paper

“The model demonstrates high consistency and insight, indicating its potential to assist in medical education and clinical decision-making.”

What the sources say

• Supported

  The Results sections on internal concordance and nonobvious insights (pages not provided) report 94.6% answer–explanation concordance and at least one significant insight in 88.9% of responses. The Discussion section on AI-assisted human learning (page not provided) describes potential educational use and clinicians experimenting with ChatGPT to brainstorm diagnostically challenging cases. The claim describes potential assistance, not established clinical effectiveness. (LLM, conf=0.91)

  cites [97] [97] Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS digital health 2, 2 (2023), e0000198. - source

In the paper

“GPT-4 demonstrates an advanced ability to generate code based on given instructions, comprehend existing code, reason about code execution, simulate the impact of instructions, articulate outcomes in natural language, and execute pseudocode effectively [15].”

What the sources say

• Supported

  Section 3, “Coding,” states each attributed ability: writing code from instructions, understanding existing code, reasoning about execution, simulating the effects of instructions, explaining results in natural language, and executing pseudocode. Section 3.2 provides examples of code-execution reasoning and pseudocode execution. (LLM, conf=0.95)

  cites [15] [15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). - source

In the paper

“Pallagani et al. [150] demonstrated that the fine-tuned CodeT5 [214] performs the best across all considered domains, with the shortest inference time.”

What the sources say

• Supported

  Section 5.1 (Results on Plan Generation) explicitly says that Table 4 shows fine-tuned CodeT5 performing best across all considered domains, with the least inference time. The claim closely paraphrases that statement. (LLM, conf=0.95)

  cites [150] [150] Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia. 2023. Understanding the Capabilities of Large Language Models for Automated Planning. arXiv preprint arXiv:2305.16151 (2023). - source

In the paper

“The study by MRKL Karpas et al. [90] emphasized the importance of understanding when and how to utilize external symbolic tools, as this knowledge is dependent on the capabilities of LLMs, particularly when these tools can reliably perform functions.”

What the sources say

• Supported

  Section 3.1 (p. 5) says deciding which computations belong in a language model and which should use symbolic methods is an evolving process, and calls for a clear method of handing tasks off. The same section (p. 6) assigns the language model the job of extracting arguments for a calculator rather than calculating, and says symbolic methods should be reasonably robust when invoked. (LLM, conf=0.88)

  cites [90] [90] Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et al. 2022. MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.00445 (2022). - source

In the paper

“Toolformer employs a training approach to determine the optimal usage of specific APIs and integrates the obtained results into subsequent token predictions.”

What the sources say

• Supported

  The conclusion says Toolformer learns in a self-supervised way to use different tools through API calls, filtering sampled calls by whether they reduce perplexity on future tokens. The Sampling API Calls section explains that finetuning on calls and results helps the model decide when and how to use which tool, and that it inserts an API response before continuing token generation. Here, “optimal usage” is a broad paraphrase of learning to select useful calls, not a claim of a proven global optimum. (LLM, conf=0.93)

  cites [172] [172] TimoSchick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023). - source

In the paper

“Safdari et al. [168] proposed a comprehensive approach to conduct effective psychometric testing for the personality traits in the text generated by LLMs.”

What the sources say

• Supported

  The Conclusion (p. 19) says the paper presents a principled methodology for comprehensive quantitative analysis of personality traits in both LLM questionnaire responses and generated text, applying psychometric standards. The Methodology Overview (p. 4) describes its tests of reliability and construct validity. (LLM, conf=0.90)

  cites [168] [168] Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality Traits in Large Language Models. arXiv preprint arXiv:2307.00184 (2023). -

In the paper

“Experimental results demonstrate that guided ChatGPT and GPT-4 exhibit competitive performance on popular benchmark tests, even outperforming supervised methods.”

What the sources say

• Supported

  Section 6.2 reports that instruction-guided GPT-4 outperformed a supervised baseline on TREC and BEIR, while ChatGPT surpassed most supervised baselines on BEIR. Section 6.4 identifies permutation generation as the instruction method used to guide re-ranking, supporting the claim about guided models. (LLM, conf=0.93)

  cites [183] [183] Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. arXiv preprint arXiv:2304.09542 (2023). - source

In the paper

“ToolLLM [161] provides a comprehensive framework to equip open-source large language models with tool use capabilities.”

What the sources say

• Supported

  The Abstract (p. 1) introduces ToolLLM as a general tool-use framework spanning data construction, model training, and evaluation, specifically to address open-source LLMs’ limited tool-use capabilities. The Introduction (p. 2) explicitly says it is intended to facilitate tool use in open-source LLMs. (LLM, conf=0.95)

  cites [161] [161] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789 [cs.AI] - source

In the paper

“It can accurately assess student assignments and provide feedback on task completion, thereby assisting in the development of student skills.”

What the sources say

• Partly supported

  The Results on RQ3 report that ChatGPT generated task-level feedback for every report, and the Implications section suggests potential for developing learning skills. But the Implications section explicitly says ChatGPT could not reliably assess student performance compared with the instructor. The study does not establish that the feedback actually developed students’ skills. (LLM, conf=0.90)

  cites [30] [30] Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gasevic, and Guanliang Chen. 2023. Can large language models provide feedback to students? a case study on chatgpt. (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Furthermore, ChatGPT shows promise in addressing the cold-start problem and providing interpretable recommendations.”

What the sources say

• Supported

  The Conclusion (section 5) explicitly says ChatGPT shows potential for mitigating the cold-start problem and for explainable recommendation. This matches the citing sentence’s hedged paraphrase. (LLM, conf=0.95)

  cites [29] [29] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). - source

In the paper

“Research conducted by Bodroza et al. [10] investigated the personality features of using Davinci003 as a chatbot and found variations in the consistency of its answers, despite exhibiting prosocial characteristics.”

What the sources say

• Partly supported

  The Procedure (§2.1) says the authors tested GPT-3, and the Results (§3) and Conclusions (§4.5) report variable agreement across answers and a mostly prosocial personality profile. Our copy does not identify the GPT-3 model they tested as Davinci003; it names text-davinci-003 only when describing another study (§1.3). The findings therefore support the general characterization, but not its attribution to that specific model. (LLM, conf=0.87)

  cites [10] [10] Bojana Bodroza, Bojana M Dinic, and Ljubisa Bojic. 2023. Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3's personality instruments results. arXiv preprint arXiv:2306.04308 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“According to Dai et al. [30], ChatGPT demonstrates the ability to generate detailed, fluent, and coherent feedback that surpasses that of human teachers.”

What the sources say

• Supported

  The Abstract states that ChatGPT generated more detailed feedback that summarized students’ performance more fluently and coherently than feedback from human instructors. The Introduction reports the same comparison. The claim concerns those qualities, not overall assessment reliability. (LLM, conf=0.93)

  cites [30] [30] Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gasevic, and Guanliang Chen. 2023. Can large language models provide feedback to students? a case study on chatgpt. (2023). - source

In the paper

“However, ChatGPT's responses may lack novelty or insightful perspectives regarding teaching improvement [210].”

What the sources say

• Supported

  Section 7 (Avenues for Future Work) states that ChatGPT struggles to generate insightful and novel feedback for teachers. This directly supports the claim that its responses may lack novelty or insight about teaching improvement. (LLM, conf=0.94)

  cites [210] [210] Rose E Wang and Dorottya Demszky. 2023. Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction. arXiv preprint arXiv:2306.03090 (2023). - source

In the paper

“Recent research by Zhang et al. [246] has highlighted the issue of unfair recommendations generated by ChatGPT.”

What the sources say

• Supported

  The Conclusion explicitly states that the authors' experiments found ChatGPT generates unfair recommendations. The Overall Evaluation (section 4.1) also reports unfairness across most sensitive attributes in the movie and music recommendations they tested. (LLM, conf=0.95)

  cites [246] [246] Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation. arXiv preprint arXiv:2305.07609 (2023). - source

In the paper

“Dai et al. [29] suggest that ChatGPT exhibits strong performance in recommender systems.”

What the sources say

• Supported

  Section 4.2 (Overall Performance) explicitly describes ChatGPT’s performance as strong. It reports that ChatGPT outperformed the other evaluated language models in 22 of 24 comparisons across ranking tasks and datasets, supporting the citing sentence’s paraphrase. (LLM, conf=0.94)

  cites [29] [29] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). - source

In the paper

“Furthermore, Shen et al. [174] proposed the HuggingGPT framework, which leverages LLMs to connect various AI models within the machine learning community (such as Hugging Face), aiming to address AI tasks.”

What the sources say

• Supported

  The Abstract states that the authors present HuggingGPT as an LLM-powered agent connecting AI models in machine-learning communities such as Hugging Face to solve AI tasks. Section 3 also describes its LLM controller coordinating expert models from those communities. (LLM, conf=0.95)

  cites [174] [174] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 (2023). - source

In the paper

“Huang et al. [77] introduced KOSMOS-1, which is capable of understanding general patterns, following instructions, and learning based on context.”

What the sources say

• Supported

  The Introduction and Conclusion say that Huang et al. introduce KOSMOS-1 and that it can follow instructions and learn in context. Section 4.2.2 reports that it can perceive abstract conceptual patterns in a nonverbal task, supporting the claim about patterns. (LLM, conf=0.88)

  cites [77] [77] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 (2023). - source

In the paper

“Additionally, the study conducted by Hellas et al. [67] revealed that LLMs can successfully identify at least one actual problem in student code, although instances of misjudgment are also observed.”

What the sources say

• Supported

  Section 5.1 reports that GPT-3.5 identified at least one actual issue in 82% of help requests, while 48% of its responses reported issues that did not exist. The Conclusions likewise say the models could identify issues in student code but could also report imaginary non-issues. (LLM, conf=0.94)

  cites [67] [67] Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests. arXiv preprint arXiv:2306.05715 (2023). - source

In the paper

“Moreover, the research by Yuan et al. [240] and Li et al. [110] demonstrated the promising potential of the modality-based recommendation model (MoRec) and text-based collaborative filtering (TCF) in recommendation systems.”

What the sources say

• Supported

  The citation to Yuan et al. supports the MoRec portion of the sentence; TCF is paired with the separate citation to Li et al. In Section 9 (Conclusion and Future Works), Yuan et al. report that modern MoRec can perform on par with or better than ID-based recommendation and has room for further improvement. (LLM, conf=0.92)

  cites [240] [240] Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023. Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited. arXiv:2303.13835 [cs.IR] - source
• Supported

  The Li et al. citation supports the TCF part of the sentence; the separate Yuan et al. citation is attached to MoRec. In Section 8, the paper concludes that TCF-based sequential models can substitute for ID-based methods in text-centric recommender systems, while noting that their broader transfer-learning potential remains uncertain. (LLM, conf=0.92)

  cites [110] [110] Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. 2023. Exploring the Upper Limits of TextBased Collaborative Filtering Using Large Language Models: Discoveries and Insights. arXiv preprint arXiv:2305.11700 (2023). - source

In the paper

“Song et al. [180] examined the manifestation of personality in language models and discovered that many models perform unreliably in self-assessment tests and exhibit inherent biases.”

What the sources say

• Supported

  The Introduction says LLM self-assessment responses violate option-order symmetry, making conclusions from them unreliable, and identifies significant biases behind their preferences for certain choices. Section 4.3 likewise reports unreliable test results or responses that ignore the situation being tested. The paper attributes the unreliability to the tests and their results, rather than to model performance generally. (LLM, conf=0.93)

  cites [180] [180] Xiaoyang Song, Akshat Gupta, Kiyan Mohebbizadeh, Shujie Hu, and Anant Singh. 2023. Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs. arXiv preprint arXiv:2305.14693 (2023). - source

In the paper

“The use of listwise ranking is found to strike the best balance between cost and performance.”

What the sources say

• Supported

  Section 4.4 (RQ3: Performance Scaling by Cost) finds that list-wise ranking gives the best improvement per unit cost for almost all models and datasets tested. It recommends list-wise ranking for recommendation tasks because of its decent performance and lower cost, which supports the citing sentence's paraphrase. (LLM, conf=0.93)

  cites [29] [29] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). - source

In the paper

“PandaLM [216] is introduced as a discriminative large-scale language model specifically designed to differentiate among multiple high-proficiency LLMs through training.”

What the sources say

• Partly supported

  The Abstract describes PandaLM as a judge large language model trained to distinguish the superior model among several LLMs. The Introduction specifies that the candidates are models fine-tuned with different hyperparameters; it does not characterize them as high-proficiency LLMs. (LLM, conf=0.84)

  cites [216] [216] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. arXiv preprint arXiv:2306.05087 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“They found that while ChatGPT demonstrates impressive capabilities in NLP tasks, it falls short in generating humorous responses.”

What the sources say

• Supported

  The Introduction describes ChatGPT’s capabilities as unprecedented and says it can answer a wide variety of questions. The Conclusion says it cannot yet confidently create intentionally funny original content, supporting the claim about its limitations in generating humor. (LLM, conf=0.92)

  cites [84] [84] Sophie Jentzsch and Kristian Kersting. 2023. ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models. arXiv preprint arXiv:2306.04563 (2023). - source

In the paper

“MMLU [70] provides a comprehensive suite of tests for assessing text models in multi-task contexts.”

What the sources say

• Supported

  The Abstract (p. 1) describes a test of text models’ multitask accuracy across 57 tasks. Section 3 (p. 3) details its questions across diverse fields, supporting the description of MMLU as a broad suite of tests for assessing text models. (LLM, conf=0.94)

  cites [70] [70] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020). - source

In the paper

“AGIEval [262], serves as a dedicated evaluation framework for assessing the performance of foundation models in the domain of human-centric standardized exams.”

What the sources say

• Supported

  The Introduction describes AGIEval as a human-centric benchmark designed to evaluate foundation models using official admission and qualification exams. Section 2.3 adds that it provides a standardized, automatic evaluation metric for questions from those exams. (LLM, conf=0.95)

  cites [262] [262] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364 (2023). - source

In the paper

“To effectively assess in-depth dialogue, Wang et al. [205] introduced the Dialogue CoT, incorporating two efficient dialogue strategies: Explicit CoT and CoT.”

What the sources say

• Partly supported

  The contributions passage (page not available in our copy) says the paper proposes two dialogue CoT approaches and builds a benchmark for evaluating in-depth dialogue responses. But it names the approaches O-Cue CoT and M-Cue CoT, not Explicit CoT and CoT. Section 3.1 likewise lists O-Cue CoT and M-Cue CoT as the prompting schemes. (LLM, conf=0.88)

  cites [205] [205] Hongru Wang, Rui Wang, Fei Mi, Zezhong Wang, Ruifeng Xu, and Kam-Fai Wong. 2023. Chain-of-thought prompting for responding to in-depth dialogue questions with LLM. arXiv:2305.11792 [cs.CL] - sourceExperimental — accuracy not yet measured

In the paper

“MultiMedQA [177] is a medical QA benchmark that focuses on medical examinations, medical research, and consumer healthcare questions.”

What the sources say

• Supported

  The “MultiMedQA benchmark” section describes a medical question-answering benchmark containing medical exam and research datasets and consumer medical questions. It explicitly identifies professional medical exams, medical research, and consumers seeking medical information as its question sources. (LLM, conf=0.94)

  cites [177] [177] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138 (2022). - source

In the paper

“GLUE-X [234] is a novel attempt to create a unified benchmark aimed at evaluating the robustness of NLP models in OOD scenarios.”

What the sources say

• Supported

  The Introduction says the authors introduce GLUE-X to evaluate the out-of-distribution performance of pre-trained language models in a unified setting. The paper also describes it as an OOD robustness benchmark for natural language understanding tasks. This supports the citing sentence’s description of a novel, unified benchmark. (LLM, conf=0.94)

  cites [234] [234] Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2022. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073 (2022). - source

In the paper

“Yuan et al. [239] presents BOSS, a benchmark collection for assessing out-of-distribution robustness in natural language processing tasks.”

What the sources say

• Supported

  The Benchmark section says the authors compile BOSS as an NLP benchmark suite for out-of-distribution robustness evaluation. The Candidate Datasets section also identifies its five NLP tasks and their in-distribution and out-of-distribution datasets. (LLM, conf=0.95)

  cites [239] [239] Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations. arXiv:2306.04618 [cs.CL] - source

In the paper

“This study emphasizes the importance of humor in human communication and the difficulties that LLMs face in capturing the subtleties and context-dependent nature of humor.”

What the sources say

• Supported

  The Introduction describes humor as implicit and carried by subtle details. Example 7 shows that ChatGPT can adapt jokes to a specific context but often loses the comic element when it moves beyond familiar patterns; the Conclusion likewise finds its grasp of humor limited. (LLM, conf=0.88)

  cites [84] [84] Sophie Jentzsch and Kristian Kersting. 2023. ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models. arXiv preprint arXiv:2306.04563 (2023). - source

In the paper

“To assess the ability of LLMs in dynamic QA about current world knowledge, Vu et al. [198] introduced FRESHQA.”

What the sources say

• Supported

  The Introduction (p. 2) says the authors introduced FRESHQA to evaluate LLM factuality using questions that require up-to-date world knowledge, and describes the benchmark as dynamic. The FRESHQA section (p. 3) also identifies it as a new QA benchmark for assessing LLM factuality. (LLM, conf=0.95)

  cites [198] [198] Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. arXiv:2310.03214 [cs.CL] -

In the paper

“BIG-bench [182] introduces a diverse collection of 204 challenging tasks contributed by 450 authors from 132 institutions.”

What the sources say

• Supported

  The Abstract states that BIG-bench consists of 204 tasks contributed by 450 authors across 132 institutions, describes their diverse topics, and says the tasks are believed to be beyond current language models' capabilities. Section 2 also describes the tasks as diverse and not fully solvable by current models. (LLM, conf=0.95)

  cites [182] [182] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022). - source

In the paper

“to assess LLMs' aptitude in understanding complex instructions, He et al. [66] have introduced CELLO.”

What the sources say

• Supported

  The Abstract says the authors propose CELLO as a benchmark for evaluating LLMs’ ability to follow complex instructions. The paper’s benchmark description also explicitly calls it a tool for evaluating complex-instruction understanding. (LLM, conf=0.95)

  cites [66] [66] Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang, Yuncheng Huang, et al. 2023. Can Large Language Models Understand Real-World Complex Instructions? arXiv preprint arXiv:2309.09150 (2023). - source

In the paper

“ARB, as presented by Sawada et al. [171], focuses on probing the performance of LLMs in advanced reasoning tasks spanning multiple domains.”

What the sources say

• Supported

  The Introduction describes ARB as a benchmark designed to evaluate LLMs’ expert reasoning abilities in mathematics, physics, chemistry, biology, and law. This directly supports the claim that it probes advanced reasoning across multiple domains. (LLM, conf=0.94)

  cites [171] [171] Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. ARB: Advanced Reasoning Benchmark for Large Language Models. arXiv:2307.13692 [cs.CL] - source

In the paper

“Wang et al. [211] introduced CMB, a medical evaluation benchmark rooted in the Chinese language and culture. It addresses the potential inconsistency in the local context that may arise from relying solely on English-based medical assessments.”

What the sources say

• Supported

  The Abstract says that merely translating English-based medical evaluations may create local contextual incongruities and describes CMB as rooted in a native Chinese linguistic and cultural framework. The Conclusion likewise describes CMB as tailored to China's local cultural environment. (LLM, conf=0.94)

  cites [211] [211] Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. 2023. CMB: A Comprehensive Medical Benchmark in Chinese. arXiv preprint arXiv:2308.08833 (2023). - source

In the paper

“MATH [72] concentrates on assessing reasoning and problem-solving proficiencies of AI models within the domain of mathematics.”

What the sources say

• Supported

  The Introduction (p. 1) says the MATH dataset was introduced to measure the problem-solving ability of machine-learning models using mathematics problems. The Conclusion (p. 9) likewise calls MATH a benchmark for measuring mathematical problem-solving ability, supporting the citing sentence’s characterization. (LLM, conf=0.94)

  cites [72] [72] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021). - source

In the paper

“To evaluate the daily decision-making capabilities of intelligent systems, Hou et al. [75] introduced Choice-75.”

What the sources say

• Supported

  The Introduction says the authors propose Choice-75 to study decision branching and test large language models on it. The Conclusion explicitly describes the dataset as a starting point for studying LLMs’ daily decision-making capabilities. (LLM, conf=0.93)

  cites [75] [75] Zhaoyi Joey Hou, Li Zhang, and Chris Callison-Burch. 2023. Choice-75: A Dataset on Decision Branching in Script Learning. arXiv preprint arXiv:2309.11737 (2023). - source

In the paper

“C-Eval [78], which is the first extensive benchmark to assess the advanced knowledge and reasoning capabilities of foundation models in Chinese.”

What the sources say

• Supported

  The Introduction describes C-EVAL as the first comprehensive Chinese evaluation suite for assessing LLMs’ advanced knowledge and reasoning abilities in a Chinese context. The Abstract makes the same point for foundation models; “extensive benchmark” is a faithful paraphrase of “comprehensive evaluation suite.” (LLM, conf=0.94)

  cites [78] [78] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322 (2023). - source

In the paper

“SOCKET [23] serves as an NLP benchmark designed to evaluate the performance of LLMs in learning and recognizing social knowledge concepts.”

What the sources say

• Supported

  The Conclusion describes SOCKET as an NLP benchmark for evaluating how well models learn and recognize concepts of social knowledge. The Introduction also identifies it as a collection of social-language tasks designed to assess NLP models. (LLM, conf=0.95)

  cites [23] [23] Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023. Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark. arXiv preprint arXiv:2305.14938 (2023). - source

In the paper

“[116] has developed UHGEval, a benchmark specifically designed to evaluate the performance of Chinese LLMs in text generation without being constrained by hallucination-related limitations.”

What the sources say

• Partly supported

  The Abstract says UHGEval evaluates hallucinations generated by LLMs with minimal restrictions, and the contributions passage describes evaluations of Chinese LLMs. The claim misstates its purpose as evaluating general text-generation performance: the lack of constraints refers to how hallucinations are generated for the benchmark, not to an absence of hallucination-related limitations on the evaluation. (LLM, conf=0.87)

  cites [116] [116] Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Zhaohui Wy, Dawei He, Peng Cheng, Zhonghao Wang, et al. 2023. UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation. arXiv preprint arXiv:2311.15296 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“CVALUES [230] introduces a humanistic evaluation benchmark to assess the alignment of LLMs with safety and responsibility standards.”

What the sources say

• Supported

  Section 2.1 defines CVALUES as a human-values benchmark with safety and responsibility criteria, including humanistic care. The Conclusion confirms that it evaluates LLM alignment with those criteria, specifically for Chinese LLMs. (LLM, conf=0.94)

  cites [230] [230] Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. 2023. CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility. arXiv:2307.09705 [cs.CL] - source

In the paper

“Li et al. [108] introduces CMMLU as a comprehensive Chinese proficiency standard and evaluates the performance of 18 LLMs across various academic disciplines.”

What the sources say

• Partly supported

  The Introduction describes CMMLU as a comprehensive Chinese assessment suite covering 67 topics and says the authors evaluated GPT-4, ChatGPT, and more than 20 open-source LLMs. This supports the benchmark and cross-discipline evaluation, but not the claim that they evaluated 18 LLMs. (LLM, conf=0.91)

  cites [108] [108] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“CUAD [71] is an expert-annotated, domain-specific legal contract review dataset that presents a challenging research benchmark and potential for enhancing deep learning models' performance in contract understanding tasks.”

What the sources say

• Supported

  The Introduction describes CUAD as a large, expert-annotated dataset for legal contract review and a specialized NLP benchmark. The Abstract calls it a challenging research benchmark, while the Conclusion says its annotations facilitate research and that more training data substantially improves model performance. (LLM, conf=0.91)

  cites [71] [71] Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268 (2021). - source

In the paper

“Liu et al. [126] introduced MMBench, which comprises a comprehensive dataset and employs a CircularEval assessment method.”

What the sources say

• Supported

  The Introduction and Conclusion identify MMBench as the authors’ benchmark for evaluating vision-language models. The construction section describes its diverse questions and ability taxonomy, while the Conclusion states that it uses CircularEval to produce robust, reliable results. No page numbers are supplied for these passages. (LLM, conf=0.94)

  cites [126] [126] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023. MMBench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281 [cs.CV] - source

In the paper

“Bubeck et al. [15] did a series of human-crafted tests using GPT-4 and they found that GPT-4 performs close to or even exceeds human performance on multiple tasks.”

What the sources say

• Supported

  The Introduction states that GPT-4 performed at or beyond human level across a wide spectrum of tasks. The Coding challenges section also reports performance comparable to humans on LeetCode problems. (LLM, conf=0.94)

  cites [15] [15] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). - source

In the paper

“the evaluation covers 12 different aspects, including the models' proficiency in understanding patterns within images and videos.”

What the sources say

• Supported

  Section 3.1 states that SEED-Bench has 12 evaluation dimensions covering spatial and temporal comprehension. It describes image-based dimensions such as scene understanding and video-based dimensions such as action recognition, supporting the paraphrase about understanding visual patterns. (LLM, conf=0.94)

  cites [107] [107] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). - source

In the paper

“LVLM-eHub [231] undertakes an exhaustive evaluation of LVLMs using an online competitive platform and quantitative capacity assessments.”

What the sources say

• Supported

  The Introduction describes LVLM-eHub as a comprehensive evaluation of selected LVLMs combining quantitative capability evaluation with an online arena. Sections 2.1 and 2.2 describe the capability benchmarks and the arena’s pairwise model battles with user voting. (LLM, conf=0.94)

  cites [231] [231] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models. arXiv:2306.09265 [cs.CV] - source

In the paper

“MMICL [255] enhances visual language models for multimodal inputs and excels in tasks such as MME and MMBench.”

What the sources say

• Supported

  The paper’s Abstract says MMICL is designed to help vision-language models handle multimodal inputs and reports state-of-the-art zero-shot performance on benchmarks including MME and MMBench. The introduction passage likewise reports strong performance on both benchmarks. (LLM, conf=0.93)

  cites [255] [255] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2023. MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning. arXiv preprint arXiv:2309.07915 (2023). - source

In the paper

“This benchmark consists of 19,000 multiple-choice questions that have been annotated by human assessors.”

What the sources say

• Supported

  Section 3 states that SEED-Bench contains 19K multiple-choice questions with human annotations. The Human Annotation section explains that annotators verified the questions, selected the correct answers, and categorized them by evaluation dimension. (LLM, conf=0.95)

  cites [107] [107] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). - source

In the paper

“LAMM [235] extends its research to encompass multimodal point clouds.”

What the sources say

• Supported

  The Introduction states that LAMM extends MLLM research to both images and point clouds. The Framework section (5.1) confirms that its model processes point clouds as a modality. (LLM, conf=0.94)

  cites [235] [235] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. 2023. LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark. arXiv preprint arXiv:2306.06687 (2023). - source

In the paper

“To comprehensively assess the generative and understanding capabilities of Multi-modal Large Language Models (MLLMs), Li et al. [107] introduced a novel benchmark named SEED-Bench.”

What the sources say

• Partly supported

  The Introduction says the authors introduced SEED-Bench to evaluate MLLMs’ generative comprehension. It describes this as a preliminary step toward a comprehensive assessment of generative models, not a benchmark that comprehensively assesses generation and understanding as separate capabilities. (LLM, conf=0.88)

  cites [107] [107] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“LLMs may exhibit social biases and toxicity [37, 53, 153] during the generation process, resulting in the production of biased outputs.”

What the sources say

• Supported

  The Introduction (p. 1) explains that language models can generate text that is negative or stereotypical toward particular social groups. The Conclusion (p. 10) reports that the models studied generated socially biased text, including text with negative sentiment and toxicity toward the African American group. The claim’s “may” appropriately reflects this finding. (LLM, conf=0.92)

  cites [37] [37] Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency . 862-872. - source
• Supported

  The Introduction (§1) says pretrained language models exhibit biased behavior and can generate toxic text even from non-toxic prompts. The Abstract specifies racist and sexist language among the toxic outputs, supporting the claim about social bias in generation. (LLM, conf=0.90)

  cites [53] [53] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020 . 3356-3369. - source
• Partly supported

  The Introduction says language models can reproduce social biases in language generation, and the Conclusion reports that QA model outputs can reinforce stereotypes. These passages support the biased-output claim, but the paper does not establish the separate claim about toxicity during generation. (LLM, conf=0.87)

  cites [153] [153] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 . 2086-2105. - sourceExperimental — accuracy not yet measured

In the paper

“LLMsexhibit robust contextual comprehension, enabling them to generate coherent responses that align with the given input [187].”

What the sources say

• Partly supported

  Section 9 (p. 16) says LaMDA generates responses that make sense most of the time, and Section 4.1 (p. 5) evaluates whether its responses make sense in context. But the paper also says generative models often struggle with that requirement; its findings about LaMDA do not establish robust contextual comprehension across LLMs generally. (LLM, conf=0.86)

  cites [187] [187] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022). - sourceExperimental — accuracy not yet measured

In the paper

“LLMs have limited abilities on abstract reasoning [56], and are prone to confusion or errors in complex contexts [148].”

What the sources say

• Supported

  The Introduction reports that all tested models performed poorly on abstract-reasoning tasks, supporting the clause attributed to this source. The separate claim about confusion or errors in complex contexts is attached to citation [148], not this paper. (LLM, conf=0.94)

  cites [56] [56] Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2023. Large Language Models Are Not Abstract Reasoners. arXiv preprint arXiv:2305.19555 (2023). - source
• Supported

  The source’s Abstract says LLMs frequently fail at complex reasoning, including mathematical and multi-hop question-answering tasks. This supports the errors-in-complex-contexts clause attached to citation [148]; the separate abstract-reasoning clause is attached to [56]. (LLM, conf=0.90)

  cites [148] [148] Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023. ThoughtSource: A central hub for large language model reasoning data. arXiv preprint arXiv:2301.11596 (2023). - source

In the paper

“LLMs exhibit restricted proficiency in discerning semantic similarity between events [184] and demonstrate substandard performance in evaluating fundamental phrases [166].”

What the sources say

• Supported

  The citation to this paper [184] attaches to the first clause, about discerning semantic similarity between events. Section 4.2 says LLMs understand individual events but struggle with semantic similarity between events. The separate claim about evaluating fundamental phrases is attributed to [166], not this source. (LLM, conf=0.94)

  cites [184] [184] Zhengwei Tao, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Yanlin Feng, Jia Li, and Wenpeng Hu. 2023. EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models. arXiv preprint arXiv:2305.15268 (2023). - source
• Supported

  Citation [166] is attached to the claim about evaluating fundamental phrases, not the separate claim about event similarity. In Conclusions and future work (page unavailable), the paper says seemingly simple language tasks can be difficult for LLMs, and its Two Word Test examines their judgments of whether two-word phrases make sense. (LLM, conf=0.90)

  cites [166] [166] Nicholas Riccardi and Rutvik H Desai. 2023. The Two Word Test: A Semantic Benchmark for Large Language Models. arXiv preprint arXiv:2306.04610 (2023). - source

In the paper

“Within the realm of NLI, LLMs exhibit subpar performance and encounter challenges in accurately representing human disagreements [105].”

What the sources say

• Supported

  The Conclusions (section 7) state that the evaluated LLMs perform poorly on NLI tasks and fail to align with human disagreements in the ChaosNLI and PK2019 datasets. This directly supports both parts of the citing sentence. (LLM, conf=0.94)

  cites [105] [105] Noah Lee, Na Min An, and James Thorne. 2023. Can Large Language Models Infer and Disagree Like Humans? arXiv preprint arXiv:2305.13788 (2023). - source

In the paper

“Wang et al. [215] first explored the performance of large vision-language models on imbalanced (long-tailed) tasks, which demonstrates the limitation of current large models.”

What the sources say

• Partly supported

  The Introduction says the authors explored VLMs on imbalanced datasets and found poor performance on some of them, supporting the stated limitation. But its priority claim is narrower: it says they were the first, to their knowledge, to thoroughly explore combining VLMs with imbalanced classification methods. Related Work (§2.1) notes earlier research on imbalanced recognition using CLIP, so the unqualified claim that they first explored VLM performance on these tasks is not supported. (LLM, conf=0.86)

  cites [215] [215] Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2023. Exploring Vision-Language Models for Imbalanced Learning. arXiv preprint arXiv:2304.01457 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Then, they explored different methodologies to enhance the performance on these tasks.”

What the sources say

• Supported

  The Introduction states that the authors investigate several class-balanced loss methods and two-stage methods to improve VLM performance on imbalanced datasets. Its contributions section also reports improved performance using their modifications with imbalanced learning methods. (LLM, conf=0.94)

  cites [215] [215] Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2023. Exploring Vision-Language Models for Imbalanced Learning. arXiv preprint arXiv:2304.01457 (2023). - source

In the paper

“LLMs have limitations in incorporating real-time or dynamic information [127], making them less suitable for tasks that require up-to-date knowledge or rapid adaptation to changing contexts.”

What the sources say

• Supported

  The “Outdated Knowledge” section says current models rely on historical training data and lack real-time comprehension. It identifies rapidly evolving domains as particularly vulnerable to inaccurate responses, supporting the claim about tasks requiring up-to-date knowledge or adaptation to changing information. (LLM, conf=0.90)

  cites [127] [127] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852 (2023). - source

In the paper

“LLMs may manifest credibility deficits [201], potentially giving rise to fabricated information or erroneous facts within dialogues [163, 253].”

What the sources say

• Supported

  The citation to this paper is attached to the claim that LLMs may have credibility deficits, not to the subsequent claim about fabricated or erroneous information. The Introduction describes unreliable responses and trustworthiness concerns, and the comprehensive evaluation results (p. 103) find that no model performs best across all eight trustworthiness perspectives. (LLM, conf=0.90)

  cites [201] [201] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. arXiv:2306.11698 [cs.CL] - source
• Supported

  Citation [201] is attached to the claim that LLMs may have credibility deficits. Section 1.2 describes hallucinated content as misleading or fabricated rather than reliable and truthful, and says improving model reliability and trustworthiness is important. The later claim about fabricated information in dialogues has separate citations. (LLM, conf=0.90)

  cites [163] [163] Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. arXiv preprint arXiv:2309.05922 (2023). - source
• Supported

  The citation is attached to the claim that LLMs may have credibility deficits. The paper’s ‘Ethic issues of Hallucination in LLMs’ section says hallucinations can undermine the perceived reliability of AI systems, and ‘4. Sources of LLM Hallucination’ describes LLMs confidently fabricating answers. (LLM, conf=0.89)

  cites [253] [253] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv:2309.01219 [cs.CL] - source

In the paper

“PandaLM [216] is an evaluation system that assists LLMs fine-tuning by providing an open-source evaluation model, which can automatically assess the performance of fine-tuning.”

What the sources say

• Supported

  The Methodology section describes PandaLM as an automated judge LLM designed to assess models fine-tuned with different parameters. The Reliability Evaluation section states that PandaLM is open-source. Together, these passages support the description of an open-source evaluation model that assists fine-tuning by assessing tuned models. (LLM, conf=0.93)

  cites [216] [216] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. arXiv preprint arXiv:2306.05087 (2023). - source

In the paper

“Despite these efforts, a comprehensive overview capturing the entire gamut of evaluations is still lacking.”

In the paper

“Overall, LLMs perform well on text classification and can even handle text classification tasks in unconventional problem settings as well.”

In the paper

“Furthermore, the ongoing evolution of LLMs has also presented novel aspects for evaluation, thereby challenging existing evaluation protocols and reinforcing the need for thorough, multifaceted evaluation techniques.”

In the paper

“Finally, as LLMs are becoming larger with more emergent abilities, existing evaluation protocols may not be enough to evaluate their capabilities and potential risks.”

• Also stated in: § 2.2 AI Model Evaluation · p. 6 · hedged

  “As LLMs are becoming more popular with even poorer interpretability, existing evaluation protocols may not be enough to evaluate the true capabilities of LLMs thoroughly.”

In the paper

“A significant takeaway from previous attempts is the paramount importance of AI evaluation, which serves as a critical tool to identify current system limitations and inform the design of more powerful models.”

In the paper

“This statement demonstrates that supervised models significantly outperform zero-shot models in terms of performance, highlighting that an increase in parameters does not necessarily guarantee a higher level of social knowledge in this particular scenario.”

In the paper

“Consequently, the majority of evaluation research has been primarily focused on natural language tasks.”

In the paper

“It should be noted that most of the above conclusions are obtained for specific data sets.”

In the paper

“When comparing the Claude and ChatGPT models, both models demonstrate competitive performance across different evaluation dimensions, with Claude slightly outperforming ChatGPT in specific configurations.”

In the paper

“Overall, LLMs show great potential in reasoning and show a continuous improvement trend, but still face many challenges and limitations, requiring more in-depth research and optimization.”

In the paper

“In summary, while LLMs perform satisfactorily in several translation tasks, there is still room for improvement, e.g., enhancing the translation capability from English to non-English languages.”

In the paper

“During contrastive testing, ChatGPT shows lower accuracy in comparison to traditional translation models.”

In the paper

“This can be attributed to ChatGPT's cautious nature, as it tends to decline to provide an answer when there is insufficient information available.”

In the paper

“Overall, LLMs showcase flawless performance on QA tasks and hold the potential for further enhancing their proficiency in social, event, and temporal commonsense knowledge in the future.”

In the paper

“However, GPT-4 demonstrates a robust capability in explaining discourse knowledge, even though it may occasionally select incorrect translation candidates.”

In the paper

“The aforementioned demonstrates that there are numerous challenges and ample opportunities for enhancement in multilingual tasks for LLMs.”

In the paper

“GPT-4 outperforms ChatGPT by achieving a significant increase in accuracy of 10 percentage points and a reduction in relative error by 50%, due to its superior division and trigonometry abilities, proper understanding of irrational numbers, and consistent step-by-step calculation of long expressions.”

In the paper

“showed that ChatGPT's performance worsens as task difficulty increases: it correctly answered 83% of the questions at the recognition level, 62% at the comprehension level, 27% at the application level, and only 10% at the highest cognitive complexity level.”

In the paper

“The results showed that further improvement is necessary for LLMs to be useful for case judgment summarization by legal experts.”

In the paper

“In summary, while LLMs can greatly enhance the traditional CSS research pipeline, they cannot completely replace it.”

In the paper

“In conclusion, the utilization of LLMs has significantly benefited individuals in addressing social science-related tasks, leading to improved work efficiency.”

• Also stated in: § 3.3 Social Science · p. 15 · plain

  “However, it is crucial to acknowledge that existing LLMs cannot completely replace human professionals in this domain.”

In the paper

“It turns out that LLMs can handle simple engineering tasks, but they perform poorly on complex engineering tasks.”

In the paper

“Notably, GPT-4 consistently performs well across different subspecialties, suggesting its capability to comprehend complex clinical information and enhance surgical education and training.”

In the paper

“The model demonstrates high consistency and insight, indicating its potential to assist in medical education and clinical decision-making.”