CoChat
Sign Up
Cited by 650Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Harnessing the power of llms in practice: a survey on chatGPT and beyond

Open access
Yang, Jingfeng, Jin, Hongye, Tang, Ruixiang, Han, Xiaotian, Feng, Qizhang, Jiang, Haoming, Zhong, Shaochen, Yin, Bing +1 more·2024·ACM Transactions on Knowledge Discovery from Data
DOI 10.1145/3649506
Open this paper in CoChat — free accountOpen PDF

Abstract

This article presents a comprehensive and practical guide for practitioners and end-users working with Large Language Models (LLMs) in their downstream Natural Language Processing (NLP) tasks. We provide discussions and insights into the usage of LLMs from the perspectives of models, data, and downstream tasks. First, we offer an introduction and brief summary of current language models. Then, we discuss the influence of pre-training data, training data, and test data. Most importantly, we provide a detailed discussion about the use and non-use cases of large language models for various natural language processing tasks, such as knowledge-intensive tasks, traditional natural language understanding tasks, generation tasks, emergent abilities, and considerations for specific tasks. We present various use cases and non-use cases to illustrate the practical applications and limitations of LLMs in real-world scenarios. We also try to understand the importance of data and the specific challenges associated with each NLP task. Furthermore, we explore the impact of spurious biases on LLMs and delve into other essential considerations, such as efficiency, cost, and latency, to ensure a comprehensive understanding of deploying LLMs in practice. This comprehensive guide aims to provide researchers and practitioners with valuable insights and best practices for working with LLMs, thereby enabling the successful implementation of these models in a wide range of NLP tasks. A curated list of practical guide resources of LLMs, regularly updated, can be found at https://github.com/Mooler0410/LLMsPracticalGuide . An LLMs evolutionary tree, editable yet regularly updated, can be found at llmtree.ai .

AI snapshot

Written from the paper’s full text.

TL;DR

This survey offers a practical guide for choosing and deploying large language models (LLMs) across NLP tasks, organized around models, data, downstream tasks, and deployment considerations. It argues that LLMs excel in generation, knowledge-intensive, out-of-distribution, and real-world scenarios, while fine-tuned models often remain preferable for traditional NLU, resource-rich or extremely low-resource translation, regression, and multimodal tasks.

Key findings

  • LLMs are generally superior for generation, knowledge-intensive QA, and generalization to out-of-distribution or noisy real-world inputs, but fine-tuned models often win on traditional NLU benchmarks with rich annotated data. For example, on CivilComments toxicity detection, even the best LLM was only better than random guessing, while fine-tuned models performed much better; on CoNLL03 NER, fine-tuned models performed around twice as well as LLMs.
  • Data availability shapes model choice: zero- and few-shot settings favor LLMs, while abundant annotated data can make fine-tuned models reasonable or preferable. Pretraining-data alignment matters; a 1.3B-parameter Transformer trained on 6B tokens of “textbook quality” web data plus 1B synthetic GPT-3.5 tokens achieved commendable code-generation performance.
  • Scaling improves many abilities, including reasoning, but is not uniformly beneficial: GPT-3’s two-digit addition only became apparent above 13B parameters, while inverse-scaling and U-shaped phenomena mean larger models can perform worse on some tasks. GPT-4 reached 86.5% on MMLU, 96.3% on ARC-C, and passed 25% of Leetcode problems.
  • Efficiency and trustworthiness are major practical constraints: GPT-3 175B required $4.6M for a single training run and 499B training tokens, T5 11B cost well over $1.3M, and PaLM training consumed about 3.4 GWh. Latency examples include GPT-J 6B inference times of 0.077 s, 0.203 s, and 0.707 s for max token sizes 2, 8, and 32, respectively.

Methods

  • Design: a comprehensive practical guide/survey, not a new empirical study; it synthesizes prior literature, benchmarks, and examples into use-case and no-use-case guidance for LLMs versus fine-tuned models.
  • Data or sample: no primary dataset or experiment is introduced; the paper draws on published results, benchmarks, and referenced systems such as GPT-3, GPT-4, PaLM, BLOOM, T5, BERT, and others.
  • Main analysis: qualitative synthesis organized by model architectures, pretraining/fine-tuning/test data, NLP task categories, scaling phenomena, efficiency, trustworthiness, safety, and practical challenges, summarized in a decision flow for choosing LLMs or fine-tuned models.

Limitations

  • Stated in the paper: real-world evaluation remains an open problem, and there is no massive systematic comparison between fine-tuned models and LLMs yet; predicting performance with scaling is also difficult, and challenges such as limited context length, inference latency, hallucinations, safety alignment, and harmful content remain unresolved.
  • Stated in the paper: in-context learning efficacy can vary with prompting template and the choice/order of examples, and its inner workings are not fully understood; LLM-based evaluators may also have a bias toward LLM-generated texts.
  • Evident from the text: as a survey, it does not report a systematic search strategy, inclusion criteria, or meta-analysis, and its recommendations are qualitative rather than validated by new controlled experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3649506

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 155 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    Fine-tuned models still have better performance than LLMs in most natural language understanding tasks, such as tasks in GLUE and SuperGLUE, if such tasks come with rich well-annotated data and contain very few out-of-distribution examples on test sets.

    “In most natural language understanding tasks, such as tasks in GLUE [116] and SuperGLUE [115], fine-tuned models still have better performance, if such tasks come with rich well-annotated data and contain very few out-of-distribution examples on test sets.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [116] source

      Not checked in the free run.

    • cites [115]

      Not checked in the free run.

  • Findingreasoning

    LLMs are particularly good at translating some low-resource language texts to English, such as in Romanian-English translation of WMT'16, where zero-shot or few-shot LLMs can perform better than SOTA fine-tuned model.

    “LLMs are particularly good at translating some low-resource language texts to English texts, such as in the Romanian-English translation of WMT'16 [13], zeroshot or few-shot LLMs can perform better than SOTA fine-tuned model [25].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [13] source

      Not checked in the free run.

    • cites [25]

      Not checked in the free run.

  • Findingnone given

    LLMs have shown superior performance on ANLI, especially on the R3 and R2.

    “LLMs have shown superior performance on ANLI, especially on the R3 and R2.”
    Quote in the paperSubstantial claim
  • Findingnone given

    In text classification, on most datasets, LLMs perform slightly worse than fine-tuned models.

    “In text classification, on most datasets, LLMs perform slightly worse than fine-tuned models.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    News articles generated by LLMs are almost indistinguishable from real news articles by humans.

    “One example is that the news articles generated by LLMs are almost indistinguishable from real news articles by humans [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • May combine two claims. 59
  • Classified as probably not a substantial claim. 87

  • May combine two claims.

    Findingcitation

    In machine translation, LLMs can perform competent translation, although the average performance is slightly worse than some commercial translation tools considering automatic metrics like BLEU.

    “In machine translation (MT), LLMs can perform competent translation, although the average performance is slightly worse than some commercial translation tools [53] considering some automatic metrics like BLEU [91].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [53]

      Not checked in the free run.

    • cites [91]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    For toxicity detection, all LLMs cannot perform well, and on CivilComments even the best one is only better than random guessing.

    “All LLMs cannot perform well on this task, and on CivilComments [15] even the best one is only better than random guessing [69].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [15]

      Not checked in the free run.

    • cites [69]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    For QA, on SQuADv2, QuAC, and many other datasets, fine-tuned models have superior performance, while on CoQA, LLMs perform as well as fine-tuned models.

    “For question answering (QA), on SQuADv2 [97], QuAC [24], and many other datasets, fine-tuned models have superior performance, while on CoQA [98], LLMs perform as well as fine-tuned models [25].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [97] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

    • cites [98] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    On NLU tasks, after fine-tuning, masked language models like T5 are better than most autoregressive language models at the same scale, while some recent results imply that this gap can be bridged by scaling.

    “Another interesting finding is that on NLU tasks, after fine-tuning, masked language models, like T5 [96], are better than most autoregressive language models at the same scale, while some recent results imply that this gap can be bridged by scaling [25].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [96]

      Not checked in the free run.

    • cites [25]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    For NER, CoNLL03 is still a challenge for LLMs, where the performance of fine-tuned models is around twice as LLMs.

    “According to available evaluation results, for the NER task, CoNLL03 [100] is still a challenge for LLMs [94], where the performance of fine-tuned models is around as twice as LLMs.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [100]

      Not checked in the free run.

    • cites [94]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    For NLI, on most datasets such as RTE and SNLI, fine-tuned models perform better than LLMs, while on CB, LLMs have obtained comparable performance with fine-tuned models.

    “For natural language inference (NLI) tasks, on most datasets, such as on RTE [116] and SNLI [16], fine-tuned models perform better than LLMs, while on some data such as CB [115], LLMs have obtained comparable performance with fine-tuned models [25].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [116] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

    • cites [115]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    On CNN/DailyMail and XSUM, fine-tuned models like Brio and Pegasus have much better ROUGE performance than any LLMs, but LLMs like OPT perform far better in human evaluation considering faithfulness, coherence, and relevance.

    “For example, on CNN/DailyMail [83] and XSUM [84], fine-tuned models like Brio [75] and Pegasus [136] have much better performance than any LLMs w.r.t.ROUGE, but LLMs like OPT [137] perform far better in human evaluation considering all aspects including faithfulness, coherence, and relevance [138].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [83] source

      Not checked in the free run.

    • cites [84]

      Not checked in the free run.

    • cites [75]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    For summarization, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics such as ROUGE, human evaluation results indicate that humans tend to prefer the results generated by LLMs.

    “For summarization tasks, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics, such as ROUGE [70], human evaluation results indicate that humans tend to prefer the results generated by LLMs [42,138] compared to that of fine-tuned models.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [70]

      Not checked in the free run.

    • cites [42]

      Not checked in the free run.

    • cites [138] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    LLMs perform quite well facing distributional differences between test/user data and training data because they do not have an explicit fitting process.

    “However, LLMs perform quite well facing such scenarios, because they do not have an explicit fitting process.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    LLMs are more versatile with respect to data availability, while fine-tuned models can be considered with abundant annotated data.

    “In a brief summary: LLMs are more versatile w.r.t.data availability, while fine-tuned models can be considered with abundant annotated data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    English resources compose the main part of the pre-training data, which mainly explains why LLMs are particularly good at translating low-resource language texts to English.

    “This is mainly due to the fact that English resources compose the main part of the pre-training data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    In-context learning does not require weight update, which mechanically prevents unfavorable fine-tuning-related behavior like catastrophic forgetting.

    “This is evidenced by LLMs impressive zero-shot and few-shot learning ability [18,132], as well as the fact that in-context learning does not require weight update, which mechanically prevents unfavorable fine-tuning-related behavior like catastrophic forgetting.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Current summarization benchmarks do not contain summaries with high quality or the automatic metrics are not proper for the evaluation of summarization.

    “However, it implies that current summarization benchmarks do not contain summaries with high quality or the automatic metrics are not proper for the evaluation of summarization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    ICL efficacy can vary based on factors like the prompting template or the choice and order of in-context examples.

    “Its efficacy can vary based on factors like the prompting template or the choice and order of in-context examples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    One-shot and few-shot performance make significant gains, even matching the performance of the SOTA fine-tuned open-domain models.

    “As reported in Reference [18], one-shot and few-shot performance make significant gains, even matching the performance of the SOTA fine-tuned open-domain models.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    CoT can enhance a model's proficiency in intricate reasoning tasks, yielding more precise and refined responses.

    “CoT can enhance a model's proficiency in intricate reasoning tasks, yielding more precise and refined responses.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    PaLM and BLOOM excel in multilingual tasks and machine translation with an abundance of multilingual pretraining data.

    “For example, PaLM [25] and BLOOM [102] excel in multilingual tasks and machine translation with an abundance of multilingual pretraining data.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [25]

      Not checked in the free run.

    • cites [102]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Emergent abilities are typically unpredictable and surprising.

    “The emergent ability is typically unpredictable and surprising, leading to tasks that emerge randomly or unexpectedly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Logical abilities of language models tend to emerge as model scales up.

    “The logical abilities of language models tend to emerge as the model scales up, such as logical deduction, logical sequence, and logic grid puzzles.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Commonsense reasoning increases gradually with model size.

    “Commonsense reasoning increases gradually with the growth of model size.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Advanced coding and concept understanding are use cases with emergent abilities of LLMs.

    “Additionally, other tasks, such as advanced coding (e.g., auto debugging, code line description), and concept understanding (e.g., novel concepts, simple Turing concepts), are also use cases with the emergent abilities of large language models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    On knowledge-intensive Big-bench tasks, some LLMs perform better than average humans and comparable to best human performance.

    “In such tasks, the performance of some LLMs is better than the average level of humans, and even comparable to the best human performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [105]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    PaLM exhibits emergent ability on ASCII word recognition and hyperbaton.

    “PaLM [25] exhibits the emergent ability on ASCII word recognition 4 and hyperbaton 5 task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25]

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    LLMs struggle when the knowledge requirements do not match their learned knowledge, or when they face tasks that only require contextual knowledge, in which case fine-tuned models can work as well as LLMs.

    “(2) LLMs struggle when the knowledge requirements do not match their learned knowledge, or when they face tasks that only require contextual knowledge, in which case fine-tuned models can work as well as LLMs.We walk through some specific examples in this category in Section 4.3.2 and point out that this issue can be reasonably mitigated with the help of knowledge retrieval pipelines.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    LLMs have predominantly been trained on text data.

    “Besides, LLMs have predominantly been trained on text data, focusing on capturing the intricacies of natural language processing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    LLMs generally struggle with some tasks due to differences in objectives and training data.

    “LLMs generally struggle with some tasks due to differences in objectives and training data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    U-shape phenomenon suggests small-scale and huge-scale models make predictions with different internal mechanisms.

    “In a certain sense, the Ushape phenomenon suggests that small-scale models and huge-scale models make predictions with different internal mechanisms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    On ARC-C, GPT-4 achieves 96.3% accuracy, close to 100%.

    “Especially on ARC-C, which contains difficult questions in science exams from grades 3 to 9, GPT-4 has been close to the performance of 100% (96.3%) [88].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [88]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    LLMs are excellent at mimicking human, data annotation and generation.

    “LLMs are excellent at mimicking human, data annotation and generation.They can also be used for quality evaluation in NLP tasks and have bonuses like interpretability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    When required knowledge is counterfactual to the real world, LLMs cannot work well and may perform worse than random guessing.

    “Another scenario is that the knowledge within LLMs about real world is useless to the task, or even the required knowledge is counterfactual to the real world.As a result, the LLMs cannot work well on such tasks.In some cases, inconsistent knowledge may even make the LLMs worse than random guessing.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    There are many exceptions where larger models do not bring better performance.

    “Although in most cases, as discussed above, larger models bring better performance, there are still many exceptions that should be considered when choosing the appropriate model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    GPT-3 shows emergent ability for word sorting and word unscrambling.

    “For example, GPT-3 [18] shows the emergent ability for word sorting, and word unscrambling tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    RLHF is used after instruction fine-tuning to adjust model input/output distribution.

    “RLHF is then used as an additional step after instruction fine-tuning to precisely adjust the model input and output distribution to achieve the same goal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Training PaLM consumed ~3.4 GWh in ~2 months.

    “Google reports that training PaLM consumed about 3.4 GWh in about two months [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Training T5 11B costs over $1.3M per run; GPT-3 175B costs $4.6M per run.

    “The cost of training an LLM is heavily influenced by its size, with estimates suggesting that training the 11B parameter variant of T5 costs well over $1.3 million for a single run, while a single training run of GPT-3 175B requires $4.6 million [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Zero-shot LLMs prohibit learning shortcuts from task-specific datasets, but LLMs still demonstrate some shortcut learning.

    “The zero-shot approach of LLMs prohibits the learning of shortcuts from task-specific datasets, which is prevalent in fine-tuned models.Nevertheless, LLMs still demonstrate a degree of shortcut learning issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    LLMs are better suited to handle real-world scenarios compared to fine-tuned models.

    “LLMs are better suited to handle real-world scenarios compared to fine-tuned models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Custom model training costs $0.03 per 1k tokens; usage costs $0.12 per 1k tokens.

    “However, for users who require custom models, training costs $0.03 per 1k tokens, while usage costs $0.12 per 1k tokens [3].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [3]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    When max token size is fixed at 32, inference time for InstructGPT model (davinci v2) is 1.969 s.

    “Additionally, when the max token size is fixed at 32, the inference time for the InstructGPT model (davinci v2) is 1.969 s.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    GPT-1, GPT-2, GPT-3 have 117M, 1.5B, 175B parameters respectively.

    “LLMs have grown increasingly larger in recent years, with models such as GPT-1, GPT-2, and GPT-3 featuring 117 million, 1.5 billion, and 175 billion parameters, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The success and popularity of LLMs like ChatGPT have confirmed their superiority to some extent.

    “Nevertheless, the huge success and popularity of LLMs such as chatGPT, have confirmed the superiority of LLMs to some extent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    In real-world deployment, performance, cost, and latency are all important considerations.

    “In real-world deployment, performance, cost, and latency are all important considerations, not just the performance of the models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    LLMs are good at mimicking humans, acting as a chatbot, and performing various tasks.

    “LLMs are particularly suitable for certain tasks.LLMs are good at mimicking humans, acting as a chatbot, and performing various kinds of tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    LLMs' performance on multimodal data remains largely unexplored.

    “As a result, their performance on multimodal data, which involves handling multiple data types such as text, images, audio, video, actions, and robotics, remains largely unexplored.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Many requests to models cannot be treated as NLP tasks.

    “Many requests to models even cannot be treated as NLP tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GPT-3 175B trained on 499B tokens.

    “Furthermore, the dataset size also scales rapidly with the size of the model, with GPT-3 175B trained on 499 billion tokens [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Human-feedback procedure plays an important role in acquiring ChatGPT's abilities.

    “The human-feedback procedure plays an important role in acquiring such abilities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    A main issue in real-world scenarios is evaluating model effectiveness.

    “One of the main issues when it comes to real-world scenarios is how to evaluate whether the model is good or not.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Diverse instruction fine-tuning is typically required for LLMs to better respond to open-ended user inputs.

    “Note that, to enable LLMs to better respond to open-ended user inputs and instructions, diverse instruction fine-tuning is still typically required to better align with real-world user input distribution and human intents of output.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    OpenAI's supercomputer with Microsoft has 285k CPU cores and 10k GPUs.

    “OpenAI has collaborated with Microsoft on a supercomputer hosted in the Microsoft Azure cloud, consisting of 285k CPU cores and 10k high-end GPUs to support the training of large models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Human feedback has shown promise in mitigating safety problems.

    “Some methods such as human feedback have shown promise in mitigating these problems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Public NLP tasks/datasets are designed for easy and automatic evaluation and cover only a small part of real-world usage.

    “The reason may be similar to reasons for fine-tuned models' inferiority: public NLP tasks/datasets are designed for easy and automatic evaluation, and they can only cover a small part of real-world usage.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Public NLP tasks/datasets are designed for easy and automatic evaluation, and they can only cover a small part of real-world usage.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Handling real-world scenarios requires coping with ambiguity, understanding context, and handling noisy input.

    “Handling such real-world scenarios requires coping with ambiguity, understanding context, and handling noisy input.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Training a 6B transformer consumes ~103.5 MWh.

    “The total energy consumption for training a transformer model with 6B parameters to completion is estimated to be around 103.5 MWh [33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Fine-tuned models are often tailored to specific tasks and may struggle to adapt to new or unexpected user requests.

    “Fine-tuned models, however, are often tailored to specific, well-defined tasks and may struggle to adapt to new or unexpected user requests.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    GPT-3 175B requires 3.14e23 Flops; T5 11B requires 3.30e22 Flops.

    “Another key metric that reflects the computing cost is Flops, with GPT-3 175B requiring 3.14×10 23 Flops, while a T5 11B model only requires 3.30 × 10 22 , which is 10 times less.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    LLMs can produce harmful or biased outputs and hallucinations that can result in severe consequences.

    “Safety concerns associated with LLMs should be given utmost importance as the potentially harmful or biased outputs, and hallucinations from LLMs can result in severe consequences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    GPT-3.5-turbo charges $0.002 per 1k tokens for chat service.

    “For users of the OpenAI API, pricing varies based on the model and usage, with options such as GPT-3.5-turbocharging $0.002 per 1k tokens for chat service.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    An additional system is often required for fine-tuned models to process unstructured context.

    “An additional system is often required as an assistant for fine-tuned models to process unstructured context, determine possible intents, and refine model responses accordingly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    ChatGPT is consistent, reliable, informative, and robust during multiple utterances with humans.

    “The LLMs-powered ChatGPTfoot_6 is surprising for its consistency, reliability, informativeness, and robustness during multiple utterances with humans.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs can act as good annotators and data generators for data augmentation.

    “LLMs can both act as a good annotator and data generator for data augmentation, such as in References [30,32,108,133,134].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [30]

      Not checked in the free run.

    • cites [32]

      Not checked in the free run.

    • cites [108]

      Not checked in the free run.

    • cites [133]

      Not checked in the free run.

    • cites [134]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    CoT reasoning ability of LLMs can show how an LLM reaches prediction, providing instance-level interpretation, while also improving performance.

    “The CoT reasoning ability of LLMs can show how an LLM reaches the prediction, which is a good interpretation on the instance level, while it also improves performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Average latency of the OpenAI API service for a single request can range from a few hundred milliseconds to several seconds.

    “The API latency can vary depending on the user's location, and the average latency of the OpenAI API service for a single request can range from a few hundred milliseconds to several seconds.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    LoRA considerably minimizes the number of parameters that require training for subsequent tasks, thereby increasing overall efficiency.

    “This approach considerably minimizes the number of parameters that require training for subsequent tasks, thereby increasing overall efficiency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    LoRA maintains the weights of the pre-trained model and incorporates low-rank matrices into every layer of the Transformer architecture.

    “As an illustration, the LoRA method maintains the weights of the pre-trained model and incorporates low-rank matrices into every layer of the Transformer architecture.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Aligning LLMs with human values, goals, and expectations remains a critical challenge.

    “In sum, aligning LLMs with human values, goals, and expectations remains a critical challenge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    LLMs, like ChatGPT, have increasingly been used for day-to-day queries due to their rising popularity.

    “LLMs, like ChatGPT, have increasingly been used for day-to-day queries due to their rising popularity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Alpaca-LoRA integrates LoRA into LLaMA-Alpaca, enabling LLaMA to run within hours on a single RTX 4090.

    “Alpaca-LoRA 22 proposes integrating Low-rank Adaptation (LoRA) into LLaMA-Alpaca, which enables runs LLaMA within hours on a single RTX 4090.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Current countermeasures include post-hoc detectors, which identify unusual tokens to distinguish generated from genuine content, and watermark schemes that modify the text generation process to embed detectable patterns.

    “Current countermeasures include post-hoc detectors, which identify unusual tokens to distinguish generated from genuine content, and watermark schemes that modify the text generation process to embed detectable patterns.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Due to the high coherence, quality, and plausibility of texts generated by LLMs, harmful contents from LLMs can cause significant harm, including hate speech, discrimination, incitement to violence, false narratives, and even social engineering attack.

    “Due to the high coherence, quality, and plausibility of texts generated by LLMs, harmful contents from LLMs can cause significant harm, including hate speech, discrimination, incitement to violence, false narratives, and even social engineering attack.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    While several methods are proposed for detecting machine-generated text, practical application presents numerous challenges, and further research is needed to address these issues.

    “In summary, while several methods are proposed for detecting machine-generated text, practical application presents numerous challenges, and further research is needed to address these issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    As LLMs become increasingly convincing and believable, users may develop an overreliance on them and trust them to provide accurate information in areas with which they are somewhat familiar.

    “As LLMs become increasingly convincing and believable, users may develop an overreliance on them and trust them to provide accurate information in areas with which they are somewhat familiar.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Large language models are not directly trained on fine-tuned datasets, which makes it less likely for them to learn shortcut features present in the fine-tuned dataset, thereby enhancing the model's generalization capabilities.

    “In contrast, large language models are not directly trained on fine-tuned datasets, which makes it less likely for them to learn shortcut features present in the fine-tuned dataset, thereby enhancing the model's generalization capabilities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    While LLMs significantly reduce the shortcut learning problem prevalent in fine-tuned models, they still exhibit some shortcut learning issues and should be approached with caution when deploying them in downstream applications.

    “In summary, while LLMs significantly reduce the shortcut learning problem prevalent in fine-tuned models, they still exhibit some shortcut learning issues and should be approached with caution when deploying them in downstream applications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The unpredictability of LLM performance as they scale accentuates the high resource costs often sunk in trial-and-error endeavors.

    “This unpredictability is not just a technical hurdle; it accentuates the high resource costs often sunk in trial-and-error endeavors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Parameter-efficient Tuning (PET) is an efficient technique to tune a small portion of model parameters while freezing most parameters of pre-trained LLMs.

    “Parameter-efficient Tuning (PET) is an efficient technique to tune a small portation of model parameters (or extra parameters) while freezing most parameters of the pre-trained LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    There is potential to misuse watermark schemes: by querying a watermarked LLM repeatedly, one can misclassify authentic human content as LLM-generated, further complicating detection.

    “Additionally, there is the potential to misuse the watermark schemes: by querying a watermarked LLM repeatedly, one can misclassify authentic human content as LLM-generated, further complicating detection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The main goal of PEFT is to greatly decrease computational and storage costs while keeping the performance of the original models.

    “The main goal of PEFT is to greatly decrease the computational and storage costs while keeping the performance of the original models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    While appending task descriptions to each input-output pair is a common strategy, striking a balance between datasets remains a challenge.

    “While appending task descriptions to each input-output pair is a common strategy, striking a balance between datasets remains a challenge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Feedback from humans plays an important role in getting rid of harmful outputs.

    “Also, in existing work, feedback from humans plays an important role in getting rid of harmful outputs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    These LLMs can have dual-use potential by providing required illicit information, leading to risks such as the proliferation of weapons and even terrorism attack planning.

    “These LLMs can also have dual-use potential by providing required illicit information, leading to risks such as the proliferation of weapons [87] and even terrorism attack planning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [87]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs have been shown to exhibit disparate treatment and impact, perpetuating societal biases and potentially leading to discrimination.

    “LLMs have been shown to exhibit disparate treatment and impact, perpetuating societal biases and potentially leading to discrimination [12,19,65,66].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [12]

      Not checked in the free run.

    • cites [19]

      Not checked in the free run.

    • cites [65]

      Not checked in the free run.

    • cites [66]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    This can be particularly dangerous if the model produces content that is entirely false or misleading, leading to incorrect decisions or actions taken based on that information.

    “This can be particularly dangerous if the model produces content that is entirely false or misleading, leading to incorrect decisions or actions taken based on that information.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    As the scale of pre-training datasets expands, individual researchers face challenges in comprehensively reviewing and ensuring the quality of entire documents.

    “As the scale of pre-training datasets expands, individual researchers face challenges in comprehensively reviewing and ensuring the quality of entire documents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The evaluation of deep learning models has been based, for the most part, on standard academic datasets like ImageNet.

    “The evaluation of deep learning models, for the most part, has been based on standard academic datasets, like ImageNet.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Two types of hallucinations can be identified: inherent hallucinations, where the generated text logically contradicts the input, and external hallucinations, where the given input does not suffice to verify the output's accuracy.

    “Two types of hallucinations can be identified: inherent hallucinations, where the generated text logically contradicts the input, and external hallucinations, where the given input does not suffice to verify the output's accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    InstructGPT davinci v2 (175B*) has been shown to outperform other models in terms of robustness.

    “InstructGPT davinci v2 (175B*) has been shown to outperform other models in terms of robustness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    For pre-trained models fine-tuned on multiple tasks, an appropriate taskmix ratio is crucial.

    “For pre-trained models fine-tuned on multiple tasks, an appropriate taskmix ratio is crucial.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The implementation of safeguards to detect and correct harmful contents can be mitigation.

    “The implementation of safeguards to detect and correct those contents can be mitigation [106].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [106]

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    LLMs are not infallible and may exhibit some shortcut learning during in-context learning.

    “However, LLMs are not infallible and may exhibit some shortcut learning during in-context learning.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    While some open-source models try to emulate proprietary ones, significant capability gaps persist.

    “Moreover, while some opensource models try to emulate proprietary ones, significant capability gaps persist.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Techniques like using models to produce synonymous content can aid adversaries, allowing them to retain the core meaning while altering the structure.

    “Techniques like using models to produce synonymous content can aid in this, allowing adversaries to retain the core meaning while altering the structure.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    This can be especially deceptive as the fluency of the output can mask these errors.

    “This can be especially deceptive as the fluency of the output can mask these errors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs are particularly susceptible to fairness issues, as significant performance disparities have been observed across demographic categories such as dialect, religion, gender, and race.

    “LLMs are particularly susceptible to fairness issues, as significant performance disparities have been observed across demographic categories such as dialect, religion, gender, and race [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Italian data protection agency declared that OpenAI illicitly gathered personal user data, leading Italy to become the first government to prohibit ChatGPT over privacy concerns.

    “The Italian data protection agency declared that OpenAI, the developer of ChatGPT, illicitly gathered personal user data, leading Italy to become the first government to prohibit ChatGPT over privacy concerns [1].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Common techniques for PET are LoRA, Prefix Tuning, and P-Tuning.

    “The common techniques for PET are LoRA [49], Prefix Tuning [68], P-Tuning [72,73].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [49]

      Not checked in the free run.

    • cites [68]

      Not checked in the free run.

    • cites [72]

      Not checked in the free run.

    • cites [73]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    There is a looming risk of models inheriting flawed beliefs, malicious intents, or even pursuing objectives misaligned with human values.

    “There is the looming risk of the models inheriting flawed beliefs, malicious intents, or even pursuing objectives misaligned with human values.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    All PFT methods can be helpful either for fine-tuning a model to a specific task or for tuning LLMs to meet special requirements like human alignment.

    “All these PFT methods can be helpful either for fine-tuning a model to a specific task or tuning LLMs to meet special requirements like human alignment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The challenge of bestowing LLMs with authentic human-like reasoning remains unresolved.

    “However, the challenge of bestowing LLMs with authentic human-like reasoning remains unresolved.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    In the context of LLMs, these approaches are mirrored in advanced prompting techniques, such as Chain-of-Thought prompting, which act as inductive biases.

    “In the context of LLMs, these approaches are mirrored in advanced prompting techniques, such as Chain-of-Thought prompting, which act as inductive biases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    External hallucinations are not necessarily incorrect, but they are still problematic because of their unverifiable nature.

    “Although external hallucinations are not necessarily incorrect, they are still problematic because of their unverifiable nature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Quantization decreases memory usage and boosts throughput by reducing the computational precision of weights and activations posttraining.

    “Quantization stands out as a technique that decreases memory usage and boosts throughput by reducing the computational precision of weights and activations posttraining.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Understanding novels or academic texts requires considering the whole content, not just a few words or sentences.

    “For example, understanding novels or academic texts is not just about a few words or sentences; one needs to consider the whole content.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Hybrid expert architectures utilize a collection of specialized modules combined with a routing network to trim down inference time.

    “Hybrid expert architectures utilize a collection of specialized modules combined with a routing network to trim down inference time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Cascading leverages models of varying sizes to strike a balance between accuracy and computational overhead.

    “Cascading leverages models of varying sizes to strike a balance between accuracy and computational overhead.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This delay arises from the model's serial token processing approach and extensive memory consumption, both because of its sheer size and the temporary states, like attention vectors, maintained during decoding.

    “Primarily, this delay arises from the model's serial token processing approach and extensive memory consumption, both because of its sheer size and the temporary states, like attention vectors, maintained during decoding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The choice of decoding strategy plays a pivotal role in influencing computational costs.

    “Moreover, the choice of decoding strategy plays a pivotal role in influencing computational costs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Deciphering a comment in a meeting transcript might see its meaning swing between sarcasm and seriousness based on preceding discussions.

    “Similarly, deciphering a comment in a meeting transcript might see its meaning swing between sarcasm and seriousness based on preceding discussions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Pruning strategically removes certain model weights without hampering performance.

    “Pruning, another post-training approach, strategically removes certain model weights without hampering performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Without a clear understanding of how these models arrive at their conclusions, ensuring their alignment with human values becomes an uphill battle.

    “Without a clear understanding of how these models arrive at their conclusions, ensuring their alignment with human values becomes an uphill battle.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Presently, the two prevalent strategies for embodied LLMs are employing LLMs as modality interfaces or forging ahead with end-to-end multi-modality models.

    “Presently, the two prevalent strategies are employing LLMs as modality interfaces or forging ahead with end-to-end multi-modality models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The quadratic scalability of the attention mechanism in Transformers exacerbates inference latency.

    “The quadratic scalability of the attention mechanism in Transformers exacerbates this.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    These alternative architectures combine computational efficiencies with commendable performance.

    “They combine computational efficiencies with commendable performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Discerning whether LLMs truly reason, by standards of planning, knowledge decomposition, and composition, is a point of contention.

    “Indeed, discerning whether LLMs truly”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    There is a push towards devising efficient attention mechanisms, such as linear nested attention, Transient Global attention, CoLT5, and Synthesizer.

    “There is a push towards devising efficient attention mechanisms, such as linear nested attention [80], Transient Global attention [44], CoLT5 [4], and Synthesizer [111].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [80]

      Not checked in the free run.

    • cites [44]

      Not checked in the free run.

    • cites [4]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Traditional decoding often introduces randomness at each step, leading to hallucinations.

    “Traditional decoding often introduces randomness at each step, leading to hallucinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Alternative architectures to the Transformer, with state-space models, convolutions, and recurrent neural networks, are emerging as contenders.

    “Alternately, there is also a discourse around alternative architectures to the Transformer, with state-space models, convolutions, and recurrent neural networks emerging as contenders.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Historically, the NLP community tackled reasoning via two main avenues: incorporating inductive biases into models and decoupling reasoning from modeling.

    “Historically, the NLP community tackled reasoning via two main avenues: incorporating inductive biases into models, exemplified by modular constructs or intermediate hidden abstractions, and decoupling reasoning from modeling, often achieved by creating executable interfaces.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The Confident Decoding approach suggests that hallucinations arise from decoding without proper attention to the input.

    “The Confident Decoding approach suggests that hallucinations arise from decoding without proper attention to the input.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Both approaches often falter in real-world scenarios due to their inherent limitations.

    “However, both approaches often falter in real-world scenarios due to their inherent limitations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Aligning LLMs that surpass human capabilities presents an even greater challenge.

    “Looking ahead, the horizon presents an even greater challenge: aligning LLMs that surpass human capabilities.”
    Quote in the paperNot a substantial claim
121 of 121

  • The paper relies on this without support.

    Without external tools, LLMs may occasionally make mistakes in basic calculations.

    “Nevertheless, it should be noted that, without the intervention of external tools, LLMs may occasionally make mistakes in performing basic calculations, although CoT prompting [125] can significantly improve LLMs' ability in calculations.”

  • The paper relies on this without support.

    LLMs have seen nearly the whole real-world text distribution during pretraining.

    “Essentially, LLMs has seen nearly the whole real-world text distribution during pretraining, making them well-suited for these scenarios.”

  • The paper relies on this without support.

    Human evaluation is complex and costly.

    “Considering the complexity and cost of human evaluation, there is no massive and systematic comparison between fine-tuned models and LLMs yet.”

  • The paper relies on this without support.

    Public NLP tasks/datasets are designed for easy and automatic evaluation, and they can only cover a small part of real-world usage.

    “The reason may be similar to reasons for fine-tuned models' inferiority: public NLP tasks/datasets are designed for easy and automatic evaluation, and they can only cover a small part of real-world usage.”

  • The paper relies on this without support.

    LLMs are now involved in sensitive areas such as healthcare, finance, and law.

    “Given that LLMs are now involved in sensitive areas such as healthcare, finance, and law, it is crucial to ensure that they are trustworthy and capable of producing reliable output.”

  • The paper relies on this without support.

    LLMs can infer and represent active attributes of text authors, like intentions, beliefs, or goals.

    “LLMs, despite being trained to predict the next word in textual corpora, can infer and represent active attributes of the text authors, like intentions, beliefs, or goals.”

  • The paper relies on this without support.

    The intricacies of human reasoning itself are nebulous.

    “The intricacies of human reasoning itself are nebulous; for instance, the discrete nature of our neuronal processes may hint at our discrete reasoning capabilities.”

  • The paper relies on this without support.

    LLMs are often too large to be run on a single user's machine.

    “As LLMs are often too large to be run on a single user's machine, companies provide LLM services via APIs.”

  • The paper relies on this without support.

    Standard academic datasets have limitations in their ability to reflect real-world performance accurately.

    “Such datasets, despite their significance in the evolution of AI, have limitations in their ability to reflect real-world performance accurately.”

  • The paper relies on this without support.

    Approximate duplicate data could adversely affect model performance.

    “Approximate duplicate data could adversely affect model performance.”

  • The paper relies on this without support.

    Including data in training sets that are similar or related to evaluation test sets could inflate performance metrics, with models possibly memorizing test data.

    “Including data in training sets that are similar or related to evaluation test sets could inflate performance metrics, with models possibly memorizing test data.”

  • The paper relies on this without support.

    Undetected personal identifiers, such as phone numbers or email addresses in pre-training datasets, raise severe privacy concerns.

    “It is worth noting that undetected personal identifiers, such as phone numbers or email addresses in pre-training datasets, raise severe privacy concerns.”

  • The paper relies on this without support.

    As LLMs burgeon in size and complexity, predicting their performance trajectory remains a conundrum.

    “The landscape of LLMs is marked by a unique challenge: as these models burgeon in size and complexity, predicting their performance trajectory remains a conundrum.”

  • The paper relies on this without support.

    Text produced by LLMs can closely resemble human-penned content.

    “The text produced by LLMs can closely resemble human-penned content, leading to concerns about misinformation spread.”

  • The paper relies on this without support.

    LLMs often produce hallucinations or generate inaccurate information.

    “Yet, their accuracy is pivotal as they often produce”

  • The paper relies on this without support.

    Limited context length hinders LLMs' ability to understand and generate extensive texts.

    “In the realm of language models, especially the large ones, a pertinent issue is their limited context length, hindering their ability to understand and generate extensive texts.”

  • The paper relies on this without support.

    LLMs often experience increased latency due to their intricate architectures.

    “LLMs often experience increased latency due to their intricate architectures.”

  • The paper relies on this without support.

    Seamlessly integrating LLMs with diverse modalities like vision, speech, knowledge bases, robotic actions, and more remains an uphill battle.

    “While LLMs excel at textual and coding tasks, seamlessly integrating them with diverse modalities like vision, speech, knowledge bases, robotic actions, and more remains an uphill battle.”

  • The paper relies on this without support.

    LLMs exhibit emergent reasoning capabilities but lag significantly behind human-like reasoning.

    “While LLMs exhibit emergent reasoning capabilities, they lag significantly behind human-like reasoning.”
19 of 19

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training a dedicated LLM for each agent is impractical due to the complexity of LLM training and the practical demands of a large variety and number of agents.

    “To achieve this goal, the ideal approach would be to train a dedicated LLM for each agent. However, considering the complexity of LLM training (Yang et al., 2024) and the practical demands of a large variety and number of agents, this approach is impractical.”

    From Enhancing memory retrieval in generative agents through LLM-trained cross attention networks (2025) · cites this paper as Yang et al., 2024

Cites

Flagged first, then by citations

Research works printed but not linked: 63

  1. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. Retrieved from https://arXiv:2108.07732
  2. Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell Jackson Kernion Andy Jones Anna Chen Anna Goldie Azalia Mirhoseini Cameron McKinnon et al. 2022. Constitutional AI: Harmlessness from AI u. Retrieved from https://arXiv:2212.08073
  3. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In Proceedings of the International Conference on Machine Learning. PMLR, 642–652.
  4. Samuel R. Bowman Jeeyoon Hyun Ethan Perez Edwin Chen Craig Pettit Scott Heiner Kamile Lukosuite Amanda Askell Andy Jones Anna Chen et al. 2022. Measuring progress on scalable oversight for large language models. Retrieved from https://arXiv:2211.03540
  5. Yupeng Chang Xu Wang Jindong Wang Yuan Wu Kaijie Zhu Hao Chen Linyi Yang Xiaoyuan Yi Cunxiang Wang Yidong Wang et al. 2023. A survey on evaluation of large language models. Retrieved from https://arXiv:2307.03109
  6. Guanzheng Chen Xin Li Zaiqiao Meng Shangsong Liang and Lidong Bing. 2023. Clex: Continuous length extrapolation for large language models. Retrieved from https://arXiv:2310.16450
  7. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. Retrieved from https://arXiv:2107.03374
  8. Xi Chen Xiao Wang Soravit Changpinyo A. J. Piergiovanni Piotr Padlewski Daniel Salz Sebastian Goodman Adam Grycner Basil Mustafa Lucas Beyer et al. 2022. Pali: A jointly-scaled multilingual language-image model. Retrieved from https://arXiv:2209.06794
  9. Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann et al. 2022. Palm: Scaling language modeling with pathways. Retrieved from https://arXiv:2204.02311
  10. Zheng Chu Jingchang Chen Qianglong Chen Weijiang Yu Tao He Haotian Wang Weihua Peng Ming Liu Bing Qin and Ting Liu. 2023. A survey of chain of thought reasoning: Advances frontiers and future. Retrieved from https://arXiv:2309.15402
  11. Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick and Oyvind Tafjord. 2018. Think you have solved question answering? try arc the ai2 reasoning challenge. Retrieved from https://arXiv:1803.05457
  12. Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training verifiers to solve math word problems. Retrieved from https://arXiv:2110.14168
  13. Haixing Dai Zhengliang Liu Wenxiong Liao Xiaoke Huang Zihao Wu Lin Zhao Wei Liu Ninghao Liu Sheng Li Dajiang Zhu et al. 2023. ChatAug: Leveraging ChatGPT for Text Data Augmentation. Retrieved from https://arXiv:2302.13007
  14. Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Retrieved from https://arXiv:1810.04805
  15. Qingxiu Dong Lei Li Damai Dai Ce Zheng Zhiyong Wu Baobao Chang Xu Sun Jingjing Xu and Zhifang Sui. 2022. A survey for in-context learning. Retrieved from https://arXiv:2301.00234
  16. Mengnan Du Fengxiang He Na Zou Dacheng Tao and Xia Hu. 2022. Shortcut learning of large language models in natural language understanding: A survey. Retrieved from https://arXiv:2208.11857
  17. Corentin Duchene Henri Jamet Pierre Guillaume and Reda Dehak. 2023. A benchmark for toxic comment classification on Civil Comments dataset. Retrieved from https://arXiv:2301.11125
  18. Jinlan Fu See-Kiong Ng Zhengbao Jiang and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. Retrieved from https://arXiv:2302.04166
  19. Tanya Goyal Junyi Jessy Li and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. Retrieved from https://arXiv:2209.12356
  20. Suriya Gunasekar Yi Zhang Jyoti Aneja Caio César Teodoro Mendes Allie Del Giorno Sivakanth Gopi Mojan Javaheripi Piero Kauffmann Gustavo de Rosa Olli Saarikivi et al. 2023. Textbooks are all you need. Retrieved from https://arXiv:2306.11644
  21. Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz and Tianlu Wang. 2023. Understanding in-context learning via supportive pretraining data. Retrieved from https://arXiv:2306.15091
  22. Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. Retrieved from https://arXiv:2009.03300
  23. Jordan Hoffmann Sebastian Borgeaud Arthur Mensch Elena Buchatskaya Trevor Cai Eliza Rutherford Diego de Las Casas Lisa Anne Hendricks Johannes Welbl Aidan Clark et al. 2022. Training compute-optimal large language models. Retrieved from https://arXiv:2203.15556
  24. Hang Hua Xingjian Li Dejing Dou Cheng-Zhong Xu and Jiebo Luo. 2022. Fine-tuning Pre-trained Language Models with Noise Stability Regularization. Retrieved from https://arXiv:2206.05658
  25. Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Chia-Yuan Chang and Xia Hu. 2023. Growlength: Accelerating llms pretraining by progressively growing training length. Retrieved from https://arXiv:2310.00576
  26. Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Zirui Liu Chia-Yuan Chang Huiyuan Chen and Xia Hu. 2024. LLM maybe LongLM: Self-extend LLM context window without tuning. Retrieved from https://arXiv:2401.01325
  27. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. Retrieved from https://arXiv:2302.14520
  28. Percy Liang Rishi Bommasani Tony Lee Dimitris Tsipras Dilara Soylu Michihiro Yasunaga Yian Zhang Deepak Narayanan Yuhuai Wu Ananya Kumar et al. 2022. Holistic evaluation of language models. Retrieved from https://arXiv:2211.09110
  29. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74–81.
  30. 10.18653/v1/2022.acl-short.8 · DOI · Paper page
  31. Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. Retrieved from https://arXiv:1907.11692
  32. Shayne Longpre Le Hou Tu Vu Albert Webson Hyung Won Chung Yi Tay Denny Zhou Quoc V. Le Barret Zoph Jason Wei et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. Retrieved from https://arXiv:2301.13688
  33. 10.18653/v1/2022.acl-long.556 · DOI · Paper page
  34. 10.5555/2002472.2002491 · DOI
  35. Bowen Peng Jeffrey Quesnelle Honglu Fan and Enrico Shippole. 2023. YaRN: Efficient context window extension of large language models. Retrieved from https://arXiv:2309.00071
  36. Chengwei Qin Aston Zhang Zhuosheng Zhang Jiaao Chen Michihiro Yasunaga and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? Retrieved from https://arXiv:2302.06476
  37. 10.5555/3455716.3455856 · DOI
  38. Sebastian Ruder, Matthew Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing tutorial. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HTL’19). 15.
  39. Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika Zaid Alyafeai Antoine Chaffin Arnaud Stiegler Teven Le Scao Arun Raja et al. 2021. Multitask prompted training enables zero-shot task generalization. Retrieved from https://arXiv:2110.08207
  40. Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ilić Daniel Hesslow Roman Castagné Alexandra Sasha Luccioni François Yvon Matthias Gallé et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. Retrieved from https://arXiv:2211.05100
  41. Lingfeng Shen Aayush Mishra and Daniel Khashabi. 2023. Do pretrained transformers really learn in-context by gradient descent? Retrieved from https://arXiv:2310.08540
  42. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1631–1642.
  43. Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid Adam Fisch Adam R. Brown Adam Santoro Aditya Gupta Adrià Garriga-Alonso et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Retrieved from https://arXiv:2206.04615
  44. Ruixiang Tang Yu-Neng Chuang and Xia Hu. 2023. The science of detecting LLM-generated texts. Retrieved from https://arXiv:2303.07205
  45. Ruixiang Tang Xiaotian Han Xiaoqian Jiang and Xia Hu. 2023. Does synthetic data generation of llms help clinical text mining? Retrieved from https://arXiv:2303.04360
  46. Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2021. Synthesizer: Rethinking self-attention for transformer models. In Proceedings of the International Conference on Machine Learning. PMLR, 10183–10192.
  47. Jonathan Uesato Nate Kushman Ramana Kumar Francis Song Noah Siegel Lisa Wang Antonia Creswell Geoffrey Irving and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. Retrieved from https://arXiv:2211.14275
  48. Jindong Wang Xixu Hu Wenxin Hou Hao Chen Runkai Zheng Yidong Wang Linyi Yang Haojun Huang Wei Ye Xiubo Geng et al. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. Retrieved from https://arXiv:2302.12095
  49. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In Proceedings of the International Conference on Machine Learning. PMLR, 22964–22984.
  50. Jason Wei Maarten Bosma Vincent Y. Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai and Quoc V. Le. 2021. Fine-tuned language models are zero-shot learners. Retrieved from https://arXiv:2109.01652
  51. Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Ed Chi Quoc Le and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. Retrieved from https://arXiv:2201.11903
  52. Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan, Xia Song, and Furu Wei. 2021. Multilingual machine translation systems from microsoft for WMT21 shared task. In Proceedings of the 6th Conference on Machine Translation. Association for Computational Linguistics, Online, 446–455. Retrieved from https://aclanthology.org/2021.wmt-1.54
  53. Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas L. Griffiths Yuan Cao and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Retrieved from https://arXiv:2305.10601
  54. Kang Min Yoo Dongju Park Jaewook Kang Sang-Woo Lee and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text augmentation. Retrieved from https://arXiv:2104.08826
  55. Jiayi Yuan Ruixiang Tang Xiaoqian Jiang and Xia Hu. 2023. LLM for patient-trial matching: Privacy-aware data augmentation towards better performance and generalizability. Retrieved from https://arXiv:2303.16756
  56. Daochen Zha Zaid Pervaiz Bhat Kwei-Herng Lai Fan Yang Zhimeng Jiang Shaochen Zhong and Xia Hu. 2023. Data-centric artificial intelligence: A survey. Retrieved from https://arXiv:2303.10158
  57. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the International Conference on Machine Learning. PMLR, 11328–11339.
  58. Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen Shuohui Chen Christopher Dewan Mona Diab Xian Li Xi Victoria Lin et al. 2022. Opt: Open pre-trained transformer language models. Retrieved from https://arXiv:2205.01068
  59. Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et al. 2023. A survey of large language models. Retrieved from https://arXiv:2303.18223
  60. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the International Conference on Machine Learning. PMLR, 12697–12706.
  61. Qihuang Zhong Liang Ding Juhua Liu Bo Du and Dacheng Tao. 2023. Can chatgpt understand too? A comparative study on chatgpt and fine-tuned BERT. Retrieved from https://arXiv:2302.10198
  62. Ce Zhou Qian Li Chen Li Jun Yu Yixin Liu Guangjing Wang Kai Zhang Cheng Ji Qiben Yan Lifang He et al. 2023. A comprehensive survey on pretrained foundation models: A history from BERT to chatgpt. Retrieved from https://arXiv:2302.09419
  63. Kaiyang Zhou Ziwei Liu Yu Qiao Tao Xiang and Chen Change Loy. 2022. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 4 (2022) 4396–4415.

Cited by

Sorted by citations

Related

Authors
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X.
Affiliations
  • Amazon (United States) (US)
  • Texas A&M University (US)
  • Rice University (US)
ORCID
Venue
ACM Transactions on Knowledge Discovery from Data
Publisher
Association for Computing Machinery (ACM)
Published
2024-04-26
Volume
18
Issue
6
Pages
1-32
ISSN
  • 1556-4681 (ISSN-L)
  • 1556-472X
DOI
10.1145/3649506
OpenAlex
W4392240262
Cited by
644
References
64
Citations by year
2026: 93 · 2025: 284 · 2024: 222 · 2023: 44 · 2013: 1
FWCI
88.40
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Knowledge Discovery from Data (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
NSF (IIS-2224843, IIS-1900990)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, ChatGPT, pre-training data, training data, test data, knowledge-intensive tasks, natural language understanding, text generation, emergent capabilities, LLM deployment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review