CoChat
Sign Up
Cited by 335Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its Crossref record; 501 of the 501 references it cites were found in a registry or an index.

A survey of GPT-3 family large language models including ChatGPT and GPT-4

Open access
Kalyan, Katikapalli Subramanyam·2024·Natural Language Processing Journal
DOI 10.1016/j.nlp.2023.100048
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are a special class of pretrained language models (PLMs) obtained by scaling model size, pretraining corpus and computation. LLMs, because of their large size and pretraining on large volumes of text data, exhibit special abilities which allow them to achieve remarkable performances without any task-specific training in many of the natural language processing tasks. The era of LLMs started with OpenAI’s GPT-3 model, and the popularity of LLMs has increased exponentially after the introduction of models like ChatGPT and GPT4. We refer to GPT-3 and its successor OpenAI models, including ChatGPT and GPT4, as GPT-3 family large language models (GLLMs). With the ever-rising popularity of GLLMs, especially in the research community, there is a strong need for a comprehensive survey which summarizes the recent research progress in multiple dimensions and can guide the research community with insightful future research directions. We start the survey paper with foundation concepts like transformers, transfer learning, self-supervised learning, pretrained language models and large language models. We then present a brief overview of GLLMs and discuss the performances of GLLMs in various downstream tasks, specific domains and multiple languages. We also discuss the data labelling and data augmentation abilities of GLLMs, the robustness of GLLMs, the effectiveness of GLLMs as evaluators, and finally, conclude with multiple insightful future research directions. To summarize, this comprehensive survey paper will serve as a good resource for both academic and industry people to stay updated with the latest research related to GLLMs.

Overview

Integrity signals

Registry

Crossref · 10.1016/j.nlp.2023.100048

Cites

501 of 501 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 501 references

Research works printed but not linked: 18

  1. Abacha, A.B., Yim, W.-w., Adams, G., Snider, N., Yetisgen-Yildiz, M., 2023. Overview of the MEDIQA-Chat 2023 Shared Tasks on the Summarization & Generation of Doctor-Patient Conversations. In: Proceedings of the 5th Clinical Natural Language Processing Workshop. pp. 503–513.
  2. Choromanski, K.M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J.Q., Mohiuddin, A., Kaiser, L., et al., 2020. Rethinking Attention with Performers. In: International Conference on Learning Representations.
  3. Dai, 2015, Semi-supervised sequence learning, Adv. Neural Inf. Process. Syst., 28
  4. Gilson, 2023, How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment, JMIR Med. Educ., 9
  5. Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Shujie, L., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al., 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. In: International Conference on Learning Representations.
  6. Lin, C.-Y., 2004. Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81.
  7. Liu, 2021, Self-supervised learning: Generative or contrastive, IEEE Trans. Knowl. Data Eng., 35, 857
  8. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al., 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In: Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
  9. Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
  10. Sennrich, R., Haddow, B., Birch, A., 2016. Improving Neural Machine Translation Models with Monolingual Data. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 86–96. · DOI · Paper page
  11. Sharma, S., Joshi, A., Mukhija, N., Zhao, Y., Bhathena, H., Singh, P., Santhanam, S., Biswas, P., 2022. Systematic review of effect of data augmentation using paraphrasing on Named entity recognition. In: NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research.
  12. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I., 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In: Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  13. Thapa, S., Naseem, U., Nasim, M., 2023. From humans to machines: can ChatGPT-like LLMs effectively replace human annotators in NLP tasks. In: Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media.
  14. Valmeekam, K., Olmo, A., Sreedharan, S., Kambhampati, S., 2022. Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change). In: NeurIPS 2022 Foundation Models for Decision Making Workshop.
  15. Vaswani, 2017, Attention is all you need, Adv. Neural Inf. Process. Syst., 30
  16. Xu, 2023, Multimodal learning with transformers: A survey, IEEE Trans. Pattern Anal. Mach. Intell., 10.1109/TPAMI.2023.3275156/mm1 · DOI
  17. Yang, 2019, Xlnet: Generalized autoregressive pretraining for language understanding, Adv. Neural Inf. Process. Syst., 32
  18. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al., 2022. GLM-130B: An Open Bilingual Pre-trained Model. In: The Eleventh International Conference on Learning Representations.

Cited by

Sorted by citations

See all 335

Related

Metadata

Authors
Kalyan, K. S.
Affiliations
Department of Public (IN)
Corresponding author
Kalyan, Katikapalli Subramanyam
ORCID
Kalyan, Katikapalli Subramanyam
Venue
Natural Language Processing Journal
Publisher
Elsevier BV
Published
2024-03
Volume
6
Pages
100048
ISSN
2949-7191 (ISSN-L)
DOI
10.1016/j.nlp.2023.100048
OpenAlex
W4389977189
Cited by
335
References
501
Citations by year
2026: 65 · 2025: 165 · 2024: 105
FWCI
31.54
Type
journal article
Language
en
Open access
Licence
Locations
Natural Language Processing Journal (publishedVersion, cc-by-nc-nd)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, ChatGPT, GPT-4, pre-trained language models, transformer models, transfer learning, self-supervised learning, downstream tasks, multilingual performance, data annotation, data augmentation, robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review