Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 251 of the 252 references it cites were found in a registry or an index; 81 of the 156 sources its claims cite were checked for support.

Large language models (LLMs): survey, technical frameworks, and future challenges

Open access
Kumar, Pranjal · 2024 · Artificial Intelligence Review

Abstract

Artificial intelligence (AI) has significantly impacted various fields. Large language models (LLMs) like GPT-4, BARD, PaLM, Megatron-Turing NLG, Jurassic-1 Jumbo etc., have contributed to our understanding and application of AI in these domains, along with natural language processing (NLP) techniques. This work provides a comprehensive overview of LLMs in the context of language modeling, word embeddings, and deep learning. It examines the application of LLMs in diverse fields including text generation, vision-language models, personalized learning, biomedicine, and code generation. The paper offers a detailed introduction and background on LLMs, facilitating a clear understanding of their fundamental ideas and concepts. Key language modeling architectures are also discussed, alongside a survey of recent works employing LLM methods for various downstream tasks across different domains. Additionally, it assesses the limitations of current approaches and highlights the need for new methodologies and potential directions for significant advancements in this field.

Overview

CoChat Summary

TL;DR

This survey reviews large language models (LLMs) across language modeling, word embeddings, deep learning architectures, applications, and open challenges. It synthesizes 53 selected studies and outlines technical frameworks, domain uses, and unresolved issues such as hallucination, computational cost, ethics, privacy, and data modality gaps.

Key findings

  • The review’s literature search identified 248 papers from IEEE Xplore, Web of Science, Google Scholar, Emerald, and Scopus; after deduplication and inclusion/exclusion, 49 papers were selected, increased to 61 via snowballing, and reduced to 53 studies after quality assessment.
  • LLMs are described as Transformer-based models with parameter counts typically ranging from 100 million to 500 billion; autoregressive models tend to have more parameters than masked models, Transformer layers typically range from 10 to 100, and input sequences typically range from 500 to 2,000 tokens before pre-training.
  • The survey covers applications including text generation, vision-language models, personalized learning, code generation and completion, and biomedicine, citing models such as GPT-4, BARD, PaLM, Megatron-Turing NLG, Jurassic-1 Jumbo, BioGPT, ESM, ProtGPT2, and ProGen.
  • Key challenges identified include hallucination, substantial computational requirements, ethical alignment and privacy concerns, limited developer influence over emergent LLM capabilities, and data deficiency across multimodal, multilingual, and graph modalities.

Methods

  • Systematic literature review following Kitchenham and Charters, using PIOC to frame research questions; the search string combined terms such as “LLM,” “Generative Language Models,” “Artificial Intelligence,” “Neural Networks,” “deep learning,” and “GPT.”
  • Inclusion/exclusion criteria focused on English peer-reviewed papers, NLP tasks, multi-task learning, and publication years 2014–2023; quality assessment scored studies 1/0 on results, empirical evidence, objectives, and references, with a quality score of 4 considered.
  • Main analysis is a narrative synthesis and comparison of LLM components, including CLMs vs. PLMs, architectures, learning methodologies, tokenization, training datasets, word embeddings (LSA, Word2Vec, GloVe, FastText, CoVe), deep learning methods (FFN, RNN, CNN, attention, KG-based models), applications, and challenges.

Limitations

  • The paper states that its scope is not all-encompassing, that emerging frameworks or theories may not be accounted for, that findings may change as the field progresses, and that it primarily concentrates on established methodologies and may not encompass recent developments.
  • Evident from the text: as a survey, it reports no primary experiments or new empirical validation; its selection is limited to English peer-reviewed studies and a 2014–2023 publication window, so non-English work, grey literature, and post-2023 developments are likely excluded.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10462-024-10888-y

Cites

251 of 252 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

58 Valid · 17 Not valid · 39 Not checked · 37 Own results and interpretations

37 more claims · 22 with their cited sources checked

Create a free account to see all 151 claims and to verify the 7 references still open.

See all claims — free account
  • Probably not a substantial claim. 93
  • It may state two separate claims as one. 37

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 189 references

Research works printed but not linked: 29

  1. Bao H, Dong L, Piao S, Wei, F BEiT: BERT Pre-Training of Image Transformers. In: International Conference on Learning Representations
  2. Bengio Y, Ducharme R, Vincent P (2000) A neural probabilistic language model. Adv Neural Inf Process Syst 13
  3. Dai Z, Yang Z, Yang Y, Carbonell J, Le QV, Salakhutdinov RT (2019) Transformer-xl: attentive language models beyond a fixed-length context · DOI · Paper page
  4. Donghan Y, Zhu C, Yang Y, Zeng M (2022) Jaket: joint pre-training of knowledge graph and language understanding. Proc AAAI Confer Artif Intell 36:11630–11638
  5. Fedus W, Zoph B, Shazeer N (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J Mach Learn Res 23(1):5232–5270
  6. Gage P (1994) A new algorithm for data compression. C Users J 12(2):23–38
  7. Keele S (2007) Guidelines for performing systematic literature reviews in software engineering (Vol. 5). Technical report, ver. 2.3 ebse technical report. ebse
  8. Lample G, Sablayrolles A, Ranzato MA, Denoyer L, Jégou H (2019) Large memory layers with product keys. Adv Neural Inf Process Syst, 32
  9. Lewis M, Ghazvininejad M, Ghosh G, Aghajanyan A, Wang S, Zettlemoyer L (2020) Pre-training via paraphrasing. Adv Neural Inf Process Syst 33:18470–18481
  10. Li J, Selvaraju R, Gotmare A, Joty S, Xiong C, Hoi SCH (2021) Align before fuse: vision and language representation learning with momentum distillation. Adv Neural Inf Process Syst 34:9694–9705
  11. Libbrecht P, Declerck T, Schlippe T, Mandl T, Schiffner D (2020) Nlp for student and teacher: concept for an ai based information literacy tutoring system. In: CIKM (workshops)
  12. Maas A, Daly RE, Pham PT, Huang D, Ng AY, Potts C (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp 142–150
  13. McCann B, Bradbury J, Xiong C, Socher R (2017) Learned in translation: Contextualized word vectors. Adv Neural Inf Process Syst 30
  14. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 26
  15. Mikolov T, Sutskever I, Deoras A, Le HS, Kombrink S, Cernocky J (2012) Subword language modeling with neural networks. Preprint (http://www.fit.vutbr.cz/imikolov/rnnlm/char.pdf) 8(67):20
  16. Qiao C, Huang B, Niu G, Li D, Dong D, He W, Yu D, Wu H (2018) A new method of region embedding for text classification. In: ICLR (poster)
  17. Radford A, Jeffrey W, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9
  18. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(1):5485–5551
  19. Ren H, Dai H, Dai B, Chen X, Yasunaga M, Sun H, Schuurmans D, Leskovec J, Zhou D. (2021) Lego: latent execution-guided reasoning for multi-hop question answering on knowledge graphs. In: International conference on machine learning. PMLR, pp 8959–8970
  20. Sahami M, Dumais S, Heckerman D, Horvitz E (1998) A Bayesian approach to filtering junk e-mail. In: Learning for text categorization: papers from the 1998 workshop, vol 62. Citeseer, pp 98–105
  21. Sutskever Ilya, Martens James, Hinton Geoffrey E (2011) Generating text with recurrent neural networks. In: Proceedings of the 28th international conference on machine learning (ICML-11), pp 1017–1024
  22. Tarasov DS (2015) Deep recurrent neural networks for multiple language aspect-based sentiment analysis of user reviews. In: Proceedings of the 21st international conference on computational linguistics dialog, vol 2, pp 53–64
  23. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Kaplan Jared D, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901
  24. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30
  25. Waldinger RJ, Lee RC (1969) Prow: a step toward automatic program writing. In: Proceedings of the 1st international joint conference on artificial intelligence, pp 241–252
  26. Wang SI, Manning CD (2012) Baselines and bigrams: simple, good sentiment and topic classification. In: Proceedings of the 50th annual meeting of the association for computational linguistics (volume 2: short papers), pp 90–94
  27. Wang B, Shang L, Lioma C, Jiang X, Yang H, Liu Q (2021) On position embeddings in bert. In: International conference on learning representations
  28. Xue H, Huang Y, Liu B, Peng H, Jianlong F, Li H, Luo J (2021) Probing inter-modality: visual parsing with self-attention for vision-and-language pre-training. Adv Neural Inf Process Syst 34:4514–4528
  29. Zhu P, Qian T (2018) Enhanced aspect level sentiment classification with auxiliary memory. In: Proceedings of the 27th international conference on computational linguistics, pp 1077–1087

Cited by

See all 395

Metadata

Authors
Kumar, P.
Affiliations
Lovely Professional University (IN)
Corresponding author
Kumar, Pranjal
Venue
Artificial Intelligence Review
Published
2024-08-18
Volume
57
Issue
10
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
OpenAlex
W4401671778
Cited by
395
References
189
Citations by year
2026: 219 · 2025: 162 · 2024: 14
FWCI
58.17
Type
journal article
Language
en
Open access
Licence
Locations
Artificial Intelligence Review (publishedVersion, cc-by-nc-nd)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, word embeddings, text generation, vision-language models, personalized learning, biomedical applications, code generation, GPT-4, deep learning, LLM limitations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large language models (LLMs): survey, technical frameworks, and future challenges. https://app.cochat.ai/papers/10.1007/s10462-024-10888-y
The paper
Kumar, P. (2024). Large language models (LLMs): survey, technical frameworks, and future challenges. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-10888-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.