CoChat
Sign Up
Cited by 651Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; none of the 155 sources its claims cite is checked for support yet.

Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

Open access
Yang, Jingfeng, Jin, Hongye, Tang, Ruixiang, Han, Xiaotian, Feng, Qizhang, Jiang, Haoming, Zhong, Shaochen, Yin, Bing, Hu, Xia·2024·ACM Transactions on Knowledge Discovery from Data
DOI 10.1145/3649506
Open this paper in CoChat — free accountOpen PDF

Abstract

This article presents a comprehensive and practical guide for practitioners and end-users working with Large Language Models (LLMs) in their downstream Natural Language Processing (NLP) tasks. We provide discussions and insights into the usage of LLMs from the perspectives of models, data, and downstream tasks. First, we offer an introduction and brief summary of current language models. Then, we discuss the influence of pre-training data, training data, and test data. Most importantly, we provide a detailed discussion about the use and non-use cases of large language models for various natural language processing tasks, such as knowledge-intensive tasks, traditional natural language understanding tasks, generation tasks, emergent abilities, and considerations for specific tasks. We present various use cases and non-use cases to illustrate the practical applications and limitations of LLMs in real-world scenarios. We also try to understand the importance of data and the specific challenges associated with each NLP task. Furthermore, we explore the impact of spurious biases on LLMs and delve into other essential considerations, such as efficiency, cost, and latency, to ensure a comprehensive understanding of deploying LLMs in practice. This comprehensive guide aims to provide researchers and practitioners with valuable insights and best practices for working with LLMs, thereby enabling the successful implementation of these models in a wide range of NLP tasks. A curated list of practical guide resources of LLMs, regularly updated, can be found at https://github.com/Mooler0410/LLMsPracticalGuide . An LLMs evolutionary tree, editable yet regularly updated, can be found at llmtree.ai .

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This survey offers a practical guide for choosing and deploying large language models (LLMs) across NLP tasks, organized around models, data, downstream tasks, and deployment considerations. It argues that LLMs excel in generation, knowledge-intensive, out-of-distribution, and real-world scenarios, while fine-tuned models often remain preferable for traditional NLU, resource-rich or extremely low-resource translation, regression, and multimodal tasks.

Key findings

  • LLMs are generally superior for generation, knowledge-intensive QA, and generalization to out-of-distribution or noisy real-world inputs, but fine-tuned models often win on traditional NLU benchmarks with rich annotated data. For example, on CivilComments toxicity detection, even the best LLM was only better than random guessing, while fine-tuned models performed much better; on CoNLL03 NER, fine-tuned models performed around twice as well as LLMs.
  • Data availability shapes model choice: zero- and few-shot settings favor LLMs, while abundant annotated data can make fine-tuned models reasonable or preferable. Pretraining-data alignment matters; a 1.3B-parameter Transformer trained on 6B tokens of “textbook quality” web data plus 1B synthetic GPT-3.5 tokens achieved commendable code-generation performance.
  • Scaling improves many abilities, including reasoning, but is not uniformly beneficial: GPT-3’s two-digit addition only became apparent above 13B parameters, while inverse-scaling and U-shaped phenomena mean larger models can perform worse on some tasks. GPT-4 reached 86.5% on MMLU, 96.3% on ARC-C, and passed 25% of Leetcode problems.
  • Efficiency and trustworthiness are major practical constraints: GPT-3 175B required $4.6M for a single training run and 499B training tokens, T5 11B cost well over $1.3M, and PaLM training consumed about 3.4 GWh. Latency examples include GPT-J 6B inference times of 0.077 s, 0.203 s, and 0.707 s for max token sizes 2, 8, and 32, respectively.

Methods

  • Design: a comprehensive practical guide/survey, not a new empirical study; it synthesizes prior literature, benchmarks, and examples into use-case and no-use-case guidance for LLMs versus fine-tuned models.
  • Data or sample: no primary dataset or experiment is introduced; the paper draws on published results, benchmarks, and referenced systems such as GPT-3, GPT-4, PaLM, BLOOM, T5, BERT, and others.
  • Main analysis: qualitative synthesis organized by model architectures, pretraining/fine-tuning/test data, NLP task categories, scaling phenomena, efficiency, trustworthiness, safety, and practical challenges, summarized in a decision flow for choosing LLMs or fine-tuned models.

Limitations

  • Stated in the paper: real-world evaluation remains an open problem, and there is no massive systematic comparison between fine-tuned models and LLMs yet; predicting performance with scaling is also difficult, and challenges such as limited context length, inference latency, hallucinations, safety alignment, and harmful content remain unresolved.
  • Stated in the paper: in-context learning efficacy can vary with prompting template and the choice/order of examples, and its inner workings are not fully understood; LLM-based evaluators may also have a bias toward LLM-generated texts.
  • Evident from the text: as a survey, it does not report a systematic search strategy, inclusion criteria, or meta-analysis, and its recommendations are qualitative rather than validated by new controlled experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3649506

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 155 cited sources checked · 0 supported · 0 flagged

Show all 115 claims

  • It may state two separate claims as one. 59
  • Probably not a substantial claim. 87
121 of 121

  • The paper relies on this without support.

    Without external tools, LLMs may occasionally make mistakes in basic calculations.

    “Nevertheless, it should be noted that, without the intervention of external tools, LLMs may occasionally make mistakes in performing basic calculations, although CoT prompting [125] can significantly improve LLMs' ability in calculations.”

  • The paper relies on this without support.

    LLMs have seen nearly the whole real-world text distribution during pretraining.

    “Essentially, LLMs has seen nearly the whole real-world text distribution during pretraining, making them well-suited for these scenarios.”

  • The paper relies on this without support.

    Human evaluation is complex and costly.

    “Considering the complexity and cost of human evaluation, there is no massive and systematic comparison between fine-tuned models and LLMs yet.”

  • The paper relies on this without support.

    Public NLP tasks/datasets are designed for easy and automatic evaluation, and they can only cover a small part of real-world usage.

    “The reason may be similar to reasons for fine-tuned models' inferiority: public NLP tasks/datasets are designed for easy and automatic evaluation, and they can only cover a small part of real-world usage.”

  • The paper relies on this without support.

    LLMs are now involved in sensitive areas such as healthcare, finance, and law.

    “Given that LLMs are now involved in sensitive areas such as healthcare, finance, and law, it is crucial to ensure that they are trustworthy and capable of producing reliable output.”

  • The paper relies on this without support.

    LLMs can infer and represent active attributes of text authors, like intentions, beliefs, or goals.

    “LLMs, despite being trained to predict the next word in textual corpora, can infer and represent active attributes of the text authors, like intentions, beliefs, or goals.”

  • The paper relies on this without support.

    The intricacies of human reasoning itself are nebulous.

    “The intricacies of human reasoning itself are nebulous; for instance, the discrete nature of our neuronal processes may hint at our discrete reasoning capabilities.”

  • The paper relies on this without support.

    LLMs are often too large to be run on a single user's machine.

    “As LLMs are often too large to be run on a single user's machine, companies provide LLM services via APIs.”

  • The paper relies on this without support.

    Standard academic datasets have limitations in their ability to reflect real-world performance accurately.

    “Such datasets, despite their significance in the evolution of AI, have limitations in their ability to reflect real-world performance accurately.”

  • The paper relies on this without support.

    Approximate duplicate data could adversely affect model performance.

    “Approximate duplicate data could adversely affect model performance.”

  • The paper relies on this without support.

    Including data in training sets that are similar or related to evaluation test sets could inflate performance metrics, with models possibly memorizing test data.

    “Including data in training sets that are similar or related to evaluation test sets could inflate performance metrics, with models possibly memorizing test data.”

  • The paper relies on this without support.

    Undetected personal identifiers, such as phone numbers or email addresses in pre-training datasets, raise severe privacy concerns.

    “It is worth noting that undetected personal identifiers, such as phone numbers or email addresses in pre-training datasets, raise severe privacy concerns.”

  • The paper relies on this without support.

    As LLMs burgeon in size and complexity, predicting their performance trajectory remains a conundrum.

    “The landscape of LLMs is marked by a unique challenge: as these models burgeon in size and complexity, predicting their performance trajectory remains a conundrum.”

  • The paper relies on this without support.

    Text produced by LLMs can closely resemble human-penned content.

    “The text produced by LLMs can closely resemble human-penned content, leading to concerns about misinformation spread.”

  • The paper relies on this without support.

    LLMs often produce hallucinations or generate inaccurate information.

    “Yet, their accuracy is pivotal as they often produce”

  • The paper relies on this without support.

    Limited context length hinders LLMs' ability to understand and generate extensive texts.

    “In the realm of language models, especially the large ones, a pertinent issue is their limited context length, hindering their ability to understand and generate extensive texts.”

  • The paper relies on this without support.

    LLMs often experience increased latency due to their intricate architectures.

    “LLMs often experience increased latency due to their intricate architectures.”

  • The paper relies on this without support.

    Seamlessly integrating LLMs with diverse modalities like vision, speech, knowledge bases, robotic actions, and more remains an uphill battle.

    “While LLMs excel at textual and coding tasks, seamlessly integrating them with diverse modalities like vision, speech, knowledge bases, robotic actions, and more remains an uphill battle.”

  • The paper relies on this without support.

    LLMs exhibit emergent reasoning capabilities but lag significantly behind human-like reasoning.

    “While LLMs exhibit emergent reasoning capabilities, they lag significantly behind human-like reasoning.”
19 of 19

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 64 references

Research works printed but not linked: 61

  1. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. Retrieved from https://arXiv:2108.07732
  2. Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell Jackson Kernion Andy Jones Anna Chen Anna Goldie Azalia Mirhoseini Cameron McKinnon et al. 2022. Constitutional AI: Harmlessness from AI u. Retrieved from https://arXiv:2212.08073
  3. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In Proceedings of the International Conference on Machine Learning. PMLR, 642–652.
  4. Samuel R. Bowman Jeeyoon Hyun Ethan Perez Edwin Chen Craig Pettit Scott Heiner Kamile Lukosuite Amanda Askell Andy Jones Anna Chen et al. 2022. Measuring progress on scalable oversight for large language models. Retrieved from https://arXiv:2211.03540
  5. Yupeng Chang Xu Wang Jindong Wang Yuan Wu Kaijie Zhu Hao Chen Linyi Yang Xiaoyuan Yi Cunxiang Wang Yidong Wang et al. 2023. A survey on evaluation of large language models. Retrieved from https://arXiv:2307.03109
  6. Guanzheng Chen Xin Li Zaiqiao Meng Shangsong Liang and Lidong Bing. 2023. Clex: Continuous length extrapolation for large language models. Retrieved from https://arXiv:2310.16450
  7. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. Retrieved from https://arXiv:2107.03374
  8. Xi Chen Xiao Wang Soravit Changpinyo A. J. Piergiovanni Piotr Padlewski Daniel Salz Sebastian Goodman Adam Grycner Basil Mustafa Lucas Beyer et al. 2022. Pali: A jointly-scaled multilingual language-image model. Retrieved from https://arXiv:2209.06794
  9. Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann et al. 2022. Palm: Scaling language modeling with pathways. Retrieved from https://arXiv:2204.02311
  10. Zheng Chu Jingchang Chen Qianglong Chen Weijiang Yu Tao He Haotian Wang Weihua Peng Ming Liu Bing Qin and Ting Liu. 2023. A survey of chain of thought reasoning: Advances frontiers and future. Retrieved from https://arXiv:2309.15402
  11. Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick and Oyvind Tafjord. 2018. Think you have solved question answering? try arc the ai2 reasoning challenge. Retrieved from https://arXiv:1803.05457
  12. Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training verifiers to solve math word problems. Retrieved from https://arXiv:2110.14168
  13. Haixing Dai Zhengliang Liu Wenxiong Liao Xiaoke Huang Zihao Wu Lin Zhao Wei Liu Ninghao Liu Sheng Li Dajiang Zhu et al. 2023. ChatAug: Leveraging ChatGPT for Text Data Augmentation. Retrieved from https://arXiv:2302.13007
  14. Qingxiu Dong Lei Li Damai Dai Ce Zheng Zhiyong Wu Baobao Chang Xu Sun Jingjing Xu and Zhifang Sui. 2022. A survey for in-context learning. Retrieved from https://arXiv:2301.00234
  15. Mengnan Du Fengxiang He Na Zou Dacheng Tao and Xia Hu. 2022. Shortcut learning of large language models in natural language understanding: A survey. Retrieved from https://arXiv:2208.11857
  16. Corentin Duchene Henri Jamet Pierre Guillaume and Reda Dehak. 2023. A benchmark for toxic comment classification on Civil Comments dataset. Retrieved from https://arXiv:2301.11125
  17. Jinlan Fu See-Kiong Ng Zhengbao Jiang and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. Retrieved from https://arXiv:2302.04166
  18. Tanya Goyal Junyi Jessy Li and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. Retrieved from https://arXiv:2209.12356
  19. Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz and Tianlu Wang. 2023. Understanding in-context learning via supportive pretraining data. Retrieved from https://arXiv:2306.15091
  20. Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. Retrieved from https://arXiv:2009.03300
  21. Jordan Hoffmann Sebastian Borgeaud Arthur Mensch Elena Buchatskaya Trevor Cai Eliza Rutherford Diego de Las Casas Lisa Anne Hendricks Johannes Welbl Aidan Clark et al. 2022. Training compute-optimal large language models. Retrieved from https://arXiv:2203.15556
  22. Hang Hua Xingjian Li Dejing Dou Cheng-Zhong Xu and Jiebo Luo. 2022. Fine-tuning Pre-trained Language Models with Noise Stability Regularization. Retrieved from https://arXiv:2206.05658
  23. Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Chia-Yuan Chang and Xia Hu. 2023. Growlength: Accelerating llms pretraining by progressively growing training length. Retrieved from https://arXiv:2310.00576
  24. Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Zirui Liu Chia-Yuan Chang Huiyuan Chen and Xia Hu. 2024. LLM maybe LongLM: Self-extend LLM context window without tuning. Retrieved from https://arXiv:2401.01325
  25. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. Retrieved from https://arXiv:2302.14520
  26. Percy Liang Rishi Bommasani Tony Lee Dimitris Tsipras Dilara Soylu Michihiro Yasunaga Yian Zhang Deepak Narayanan Yuhuai Wu Ananya Kumar et al. 2022. Holistic evaluation of language models. Retrieved from https://arXiv:2211.09110
  27. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74–81.
  28. 10.18653/v1/2022.acl-short.8 · DOI · Paper page
  29. Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. Retrieved from https://arXiv:1907.11692
  30. Shayne Longpre Le Hou Tu Vu Albert Webson Hyung Won Chung Yi Tay Denny Zhou Quoc V. Le Barret Zoph Jason Wei et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. Retrieved from https://arXiv:2301.13688
  31. 10.18653/v1/2022.acl-long.556 · DOI · Paper page
  32. 10.5555/2002472.2002491 · DOI
  33. Bowen Peng Jeffrey Quesnelle Honglu Fan and Enrico Shippole. 2023. YaRN: Efficient context window extension of large language models. Retrieved from https://arXiv:2309.00071
  34. Chengwei Qin Aston Zhang Zhuosheng Zhang Jiaao Chen Michihiro Yasunaga and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? Retrieved from https://arXiv:2302.06476
  35. 10.5555/3455716.3455856 · DOI
  36. Sebastian Ruder, Matthew Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing tutorial. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HTL’19). 15.
  37. Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika Zaid Alyafeai Antoine Chaffin Arnaud Stiegler Teven Le Scao Arun Raja et al. 2021. Multitask prompted training enables zero-shot task generalization. Retrieved from https://arXiv:2110.08207
  38. Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ilić Daniel Hesslow Roman Castagné Alexandra Sasha Luccioni François Yvon Matthias Gallé et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. Retrieved from https://arXiv:2211.05100
  39. Lingfeng Shen Aayush Mishra and Daniel Khashabi. 2023. Do pretrained transformers really learn in-context by gradient descent? Retrieved from https://arXiv:2310.08540
  40. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1631–1642.
  41. Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid Adam Fisch Adam R. Brown Adam Santoro Aditya Gupta Adrià Garriga-Alonso et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Retrieved from https://arXiv:2206.04615
  42. Ruixiang Tang Yu-Neng Chuang and Xia Hu. 2023. The science of detecting LLM-generated texts. Retrieved from https://arXiv:2303.07205
  43. Ruixiang Tang Xiaotian Han Xiaoqian Jiang and Xia Hu. 2023. Does synthetic data generation of llms help clinical text mining? Retrieved from https://arXiv:2303.04360
  44. Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2021. Synthesizer: Rethinking self-attention for transformer models. In Proceedings of the International Conference on Machine Learning. PMLR, 10183–10192.
  45. Jonathan Uesato Nate Kushman Ramana Kumar Francis Song Noah Siegel Lisa Wang Antonia Creswell Geoffrey Irving and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. Retrieved from https://arXiv:2211.14275
  46. Jindong Wang Xixu Hu Wenxin Hou Hao Chen Runkai Zheng Yidong Wang Linyi Yang Haojun Huang Wei Ye Xiubo Geng et al. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. Retrieved from https://arXiv:2302.12095
  47. Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In Proceedings of the International Conference on Machine Learning. PMLR, 22964–22984.
  48. Jason Wei Maarten Bosma Vincent Y. Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai and Quoc V. Le. 2021. Fine-tuned language models are zero-shot learners. Retrieved from https://arXiv:2109.01652
  49. Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Ed Chi Quoc Le and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. Retrieved from https://arXiv:2201.11903
  50. Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan, Xia Song, and Furu Wei. 2021. Multilingual machine translation systems from microsoft for WMT21 shared task. In Proceedings of the 6th Conference on Machine Translation. Association for Computational Linguistics, Online, 446–455. Retrieved from https://aclanthology.org/2021.wmt-1.54
  51. Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas L. Griffiths Yuan Cao and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Retrieved from https://arXiv:2305.10601
  52. Kang Min Yoo Dongju Park Jaewook Kang Sang-Woo Lee and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text augmentation. Retrieved from https://arXiv:2104.08826
  53. Jiayi Yuan Ruixiang Tang Xiaoqian Jiang and Xia Hu. 2023. LLM for patient-trial matching: Privacy-aware data augmentation towards better performance and generalizability. Retrieved from https://arXiv:2303.16756
  54. Daochen Zha Zaid Pervaiz Bhat Kwei-Herng Lai Fan Yang Zhimeng Jiang Shaochen Zhong and Xia Hu. 2023. Data-centric artificial intelligence: A survey. Retrieved from https://arXiv:2303.10158
  55. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the International Conference on Machine Learning. PMLR, 11328–11339.
  56. Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen Shuohui Chen Christopher Dewan Mona Diab Xian Li Xi Victoria Lin et al. 2022. Opt: Open pre-trained transformer language models. Retrieved from https://arXiv:2205.01068
  57. Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et al. 2023. A survey of large language models. Retrieved from https://arXiv:2303.18223
  58. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the International Conference on Machine Learning. PMLR, 12697–12706.
  59. Qihuang Zhong Liang Ding Juhua Liu Bo Du and Dacheng Tao. 2023. Can chatgpt understand too? A comparative study on chatgpt and fine-tuned BERT. Retrieved from https://arXiv:2302.10198
  60. Ce Zhou Qian Li Chen Li Jun Yu Yixin Liu Guangjing Wang Kai Zhang Cheng Ji Qiben Yan Lifang He et al. 2023. A comprehensive survey on pretrained foundation models: A history from BERT to chatgpt. Retrieved from https://arXiv:2302.09419
  61. Kaiyang Zhou Ziwei Liu Yu Qiao Tao Xiang and Chen Change Loy. 2022. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 4 (2022) 4396–4415.

Cited by

Sorted by citations

Items 601–651 of 651

Related

See all

Metadata

Authors
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X.
Affiliations
  • Amazon (United States) (US)
  • Texas A&M University (US)
  • Rice University (US)
ORCID
Venue
ACM Transactions on Knowledge Discovery from Data
Publisher
Association for Computing Machinery (ACM)
Published
2024-04-26
Volume
18
Issue
6
Pages
1-32
ISSN
  • 1556-4681 (ISSN-L)
  • 1556-472X
DOI
10.1145/3649506
OpenAlex
W4392240262
Cited by
644
References
64
Citations by year
2026: 93 · 2025: 284 · 2024: 222
FWCI
88.40
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Knowledge Discovery from Data (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
NSF (IIS-2224843, IIS-1900990)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, ChatGPT, pre-training data, training data, test data, knowledge-intensive tasks, natural language understanding, text generation, emergent capabilities, LLM deployment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review