The DOI matches its Crossref record; 251 of the 252 references it cites were found in a registry or an index; 81 of the 156 sources its claims cite were checked for support.
Large language models (LLMs): survey, technical frameworks, and future challenges
Open accessAbstract
Artificial intelligence (AI) has significantly impacted various fields. Large language models (LLMs) like GPT-4, BARD, PaLM, Megatron-Turing NLG, Jurassic-1 Jumbo etc., have contributed to our understanding and application of AI in these domains, along with natural language processing (NLP) techniques. This work provides a comprehensive overview of LLMs in the context of language modeling, word embeddings, and deep learning. It examines the application of LLMs in diverse fields including text generation, vision-language models, personalized learning, biomedicine, and code generation. The paper offers a detailed introduction and background on LLMs, facilitating a clear understanding of their fundamental ideas and concepts. Key language modeling architectures are also discussed, alongside a survey of recent works employing LLM methods for various downstream tasks across different domains. Additionally, it assesses the limitations of current approaches and highlights the need for new methodologies and potential directions for significant advancements in this field.
Overview
CoChat Summary
TL;DR
This survey reviews large language models (LLMs) across language modeling, word embeddings, deep learning architectures, applications, and open challenges. It synthesizes 53 selected studies and outlines technical frameworks, domain uses, and unresolved issues such as hallucination, computational cost, ethics, privacy, and data modality gaps.
Key findings
- The review’s literature search identified 248 papers from IEEE Xplore, Web of Science, Google Scholar, Emerald, and Scopus; after deduplication and inclusion/exclusion, 49 papers were selected, increased to 61 via snowballing, and reduced to 53 studies after quality assessment.
- LLMs are described as Transformer-based models with parameter counts typically ranging from 100 million to 500 billion; autoregressive models tend to have more parameters than masked models, Transformer layers typically range from 10 to 100, and input sequences typically range from 500 to 2,000 tokens before pre-training.
- The survey covers applications including text generation, vision-language models, personalized learning, code generation and completion, and biomedicine, citing models such as GPT-4, BARD, PaLM, Megatron-Turing NLG, Jurassic-1 Jumbo, BioGPT, ESM, ProtGPT2, and ProGen.
- Key challenges identified include hallucination, substantial computational requirements, ethical alignment and privacy concerns, limited developer influence over emergent LLM capabilities, and data deficiency across multimodal, multilingual, and graph modalities.
Methods
- Systematic literature review following Kitchenham and Charters, using PIOC to frame research questions; the search string combined terms such as “LLM,” “Generative Language Models,” “Artificial Intelligence,” “Neural Networks,” “deep learning,” and “GPT.”
- Inclusion/exclusion criteria focused on English peer-reviewed papers, NLP tasks, multi-task learning, and publication years 2014–2023; quality assessment scored studies 1/0 on results, empirical evidence, objectives, and references, with a quality score of 4 considered.
- Main analysis is a narrative synthesis and comparison of LLM components, including CLMs vs. PLMs, architectures, learning methodologies, tokenization, training datasets, word embeddings (LSA, Word2Vec, GloVe, FastText, CoVe), deep learning methods (FFN, RNN, CNN, attention, KG-based models), applications, and challenges.
Limitations
- The paper states that its scope is not all-encompassing, that emerging frameworks or theories may not be accounted for, that findings may change as the field progresses, and that it primarily concentrates on established methodologies and may not encompass recent developments.
- Evident from the text: as a survey, it reports no primary experiments or new empirical validation; its selection is limited to English peer-reviewed studies and a 2014–2023 publication window, so non-English work, grey literature, and post-2023 developments are likely excluded.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
58 Valid · 17 Not valid · 39 Not checked · 37 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“However, akin to question answering (QA), modality interaction is frequently confined to feeding knowledge into language (Zhang et al. 2019; Shen et al. 2020; Donghan et al. 2022), rather than fostering interactions across multiple layers.”
What the sources say
- Disputedcites Zhang et al. 2019 sourceExperimental — accuracy not yet measured
Section 3.3 describes ERNIE’s knowledgeable encoder as stacked aggregators that mutually integrate token and entity information. Figure 2 says the fusion layer outputs both new token and entity embeddings for the next layer. ERNIE therefore does not exemplify interaction confined to feeding knowledge into language without interaction across layers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Partly supportedcites Shen et al. 2020 sourceExperimental — accuracy not yet measured
The Abstract and Introduction describe injecting structured knowledge into a language model during pre-training, supporting this paper as an example of knowledge feeding into language. They do not establish that this is frequently the limit of modality interaction or compare the approach with interactions across multiple layers. Page numbers are unavailable in our copy. (LLM, conf=0.82)
- Not matchedcites Donghan et al. 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Since 2018, the fundamental structure of the Transformer language model has scarcely changed (Radford et al. 2018; Devlin et al. 2018).”
What the sources say
- Not matchedcites Radford et al. 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Devlin et al. 2018 sourceExperimental — accuracy not yet measured
Section 3 (p. 3) says BERT’s 2018 Transformer implementation was “almost identical” to the original architecture, supporting continuity up to that point. The paper does not address whether the fundamental structure scarcely changed after 2018. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Although there are comprehensive analyses (Min et al. 2021; Qiu et al. 2020) of Pretrained Foundation Models (PFMs) and their application to various NLP tasks, these analyses may not be directly applicable to LLMs due to inherent differences.”
What the sources say
- Supportedcites Min et al. 2021 source
The citation identifies Min et al. as a comprehensive analysis of pre-trained language models and their use in NLP tasks. The Conclusion describes the paper as a survey offering a comprehensive view of their training, adaptation, and use; §2.4 addresses their application to disparate NLP tasks. The statement about applicability to LLMs is the citing authors’ own qualification, not a finding attributed to this source. (LLM, conf=0.92)
- Supportedcites Qiu et al. 2020 source
The survey’s opening summary (p. 1) calls itself a comprehensive review of pre-trained models for NLP and says it covers adapting them to downstream tasks. The citation supports the existence and scope of that analysis; the statement about applicability to LLMs is the citing authors’ own assessment. (LLM, conf=0.90)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This phenomenon occurs due to the model's probabilistic nature, which predicts the next word or phrase based on learned patterns rather than a strict verification of factual accuracy (Manakul et al. 2023).”
What the sources say
- Partly supportedcites Manakul et al. 2023 sourceExperimental — accuracy not yet measured
Section 3.1 (p. 3) says LLMs are trained on next-word prediction and that uncertainty during generation is likely to produce non-factual words. It does not establish that this is the general cause of hallucinations or explicitly say that models generate text instead of verifying its factual accuracy; Section 2.3 (p. 2) describes fact verification as a separate approach. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Scholars have noted that augmenting the model's size and the quantity of training data consistently augments the model's capability, conforming to a scaling law (Kaplan et al. 2020).”
What the sources say
- Supportedcites Kaplan et al. 2020
The Summary (p. 3) says performance improves predictably when model size and dataset size are scaled together. The Summary of Scaling Laws (p. 5) describes power-law performance improvements as either is scaled up, within the studied range of language models. (LLM, conf=0.93)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
37 more claims · 22 with their cited sources checked
Create a free account to see all 151 claims and to verify the 7 references still open.
- Probably not a substantial claim. 93
- It may state two separate claims as one. 37
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“It includes 2,000,000 frequent crawl words, each represented by 300 dimensions, resulting in 600,000,000 word-vectors.”
Reported 600000000 · recomputed 2000000
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Linguistic units · p. 16
“Various tokenization strategies possess unique benefits and constraints, prompting researchers to explore different methods to identify the most appropriate one for their particular application (Table 5).”
§ 5 Word embeddings · p. 17
“Table 5 An overview of recently developed LLMs and their specifications”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1 Introduction · p. 3
“the latest GPT model, GPT-4, incorporates enhanced features”
§ 6 Deep learning for LLM tasks · p. 23
“OpenAI's latest GPT-3 model (Brown et al. 2020) continues in the same trajectory”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 Latent semantic analysis (LSA) · p. 19
“On the major diagonal, the eigenvalues have been arranged from least to greatest (Ramezani et al. 2023).”
§ 5.1 Latent semantic analysis (LSA) · p. 19
“u1D70E 1 ≥ u1D70E 2 ≥ u1D70E 3 ≥ ⋯ ≥ u1D70E r ≥ u1D70E r + 1 = ⋯ = u1D70E s = 0”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
Research works printed but not linked: 29
- Bao H, Dong L, Piao S, Wei, F BEiT: BERT Pre-Training of Image Transformers. In: International Conference on Learning Representations
- Bengio Y, Ducharme R, Vincent P (2000) A neural probabilistic language model. Adv Neural Inf Process Syst 13
- Dai Z, Yang Z, Yang Y, Carbonell J, Le QV, Salakhutdinov RT (2019) Transformer-xl: attentive language models beyond a fixed-length context · DOI · Paper page
- Donghan Y, Zhu C, Yang Y, Zeng M (2022) Jaket: joint pre-training of knowledge graph and language understanding. Proc AAAI Confer Artif Intell 36:11630–11638
- Fedus W, Zoph B, Shazeer N (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J Mach Learn Res 23(1):5232–5270
- Gage P (1994) A new algorithm for data compression. C Users J 12(2):23–38
- Keele S (2007) Guidelines for performing systematic literature reviews in software engineering (Vol. 5). Technical report, ver. 2.3 ebse technical report. ebse
- Lample G, Sablayrolles A, Ranzato MA, Denoyer L, Jégou H (2019) Large memory layers with product keys. Adv Neural Inf Process Syst, 32
- Lewis M, Ghazvininejad M, Ghosh G, Aghajanyan A, Wang S, Zettlemoyer L (2020) Pre-training via paraphrasing. Adv Neural Inf Process Syst 33:18470–18481
- Li J, Selvaraju R, Gotmare A, Joty S, Xiong C, Hoi SCH (2021) Align before fuse: vision and language representation learning with momentum distillation. Adv Neural Inf Process Syst 34:9694–9705
- Libbrecht P, Declerck T, Schlippe T, Mandl T, Schiffner D (2020) Nlp for student and teacher: concept for an ai based information literacy tutoring system. In: CIKM (workshops)
- Maas A, Daly RE, Pham PT, Huang D, Ng AY, Potts C (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp 142–150
- McCann B, Bradbury J, Xiong C, Socher R (2017) Learned in translation: Contextualized word vectors. Adv Neural Inf Process Syst 30
- Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 26
- Mikolov T, Sutskever I, Deoras A, Le HS, Kombrink S, Cernocky J (2012) Subword language modeling with neural networks. Preprint (http://www.fit.vutbr.cz/imikolov/rnnlm/char.pdf) 8(67):20
- Qiao C, Huang B, Niu G, Li D, Dong D, He W, Yu D, Wu H (2018) A new method of region embedding for text classification. In: ICLR (poster)
- Radford A, Jeffrey W, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9
- Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(1):5485–5551
- Ren H, Dai H, Dai B, Chen X, Yasunaga M, Sun H, Schuurmans D, Leskovec J, Zhou D. (2021) Lego: latent execution-guided reasoning for multi-hop question answering on knowledge graphs. In: International conference on machine learning. PMLR, pp 8959–8970
- Sahami M, Dumais S, Heckerman D, Horvitz E (1998) A Bayesian approach to filtering junk e-mail. In: Learning for text categorization: papers from the 1998 workshop, vol 62. Citeseer, pp 98–105
- Sutskever Ilya, Martens James, Hinton Geoffrey E (2011) Generating text with recurrent neural networks. In: Proceedings of the 28th international conference on machine learning (ICML-11), pp 1017–1024
- Tarasov DS (2015) Deep recurrent neural networks for multiple language aspect-based sentiment analysis of user reviews. In: Proceedings of the 21st international conference on computational linguistics dialog, vol 2, pp 53–64
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Kaplan Jared D, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901
- Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30
- Waldinger RJ, Lee RC (1969) Prow: a step toward automatic program writing. In: Proceedings of the 1st international joint conference on artificial intelligence, pp 241–252
- Wang SI, Manning CD (2012) Baselines and bigrams: simple, good sentiment and topic classification. In: Proceedings of the 50th annual meeting of the association for computational linguistics (volume 2: short papers), pp 90–94
- Wang B, Shang L, Lioma C, Jiang X, Yang H, Liu Q (2021) On position embeddings in bert. In: International conference on learning representations
- Xue H, Huang Y, Liu B, Peng H, Jianlong F, Li H, Luo J (2021) Probing inter-modality: visual parsing with self-attention for vision-and-language pre-training. Adv Neural Inf Process Syst 34:4514–4528
- Zhu P, Qian T (2018) Enhanced aspect level sentiment classification with auxiliary memory. In: Proceedings of the 27th international conference on computational linguistics, pp 1077–1087
Cited by
Unveiling trust in AI: the interplay of antecedents, consequences, and cultural dynamics
Dang, Q., & Li, G. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02477-6 · Cited by 98
“Since the early 1990s, AI technologies have evolved from rule-based expert systems to data-driven machine learning models and, more recently, to large language models (LLMs) capable of natural language understanding and generation (Kumar 2024).”
A comprehensive survey on integrating large language models with knowledge-based methods
Yang, W., Some, L., Bain, M., & Kang, B. H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113503 · Cited by 154
Multi-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hassan, S. Z., Shutske, J., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., Meng, Z., et al. (2024) · doi:10.36227/techrxiv.172651082.24507804/v1 · Cited by 68
Yilmaz, B. E., Yılmaz, B., & Özbey, F. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05926-2 · Cited by 86
“These models belong to the category of large language models (LLMs) and possess the ability to analyze and respond to complex questions through natural language processing (NLP) algorithms [13].”
Multi-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hadi, M. U., Hassan, S. Z., Sadak, F., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., et al. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3612154 · Cited by 106
“• Statistical Methods (1990s–2010s): This era introduced probabilistic models like n-grams, which predict the likelihood of a sequence of words based on the frequencies of previous word sequences in a text [53].”
Metadata
- Authors
- Kumar, P.
- Affiliations
- Lovely Professional University (IN)
- Corresponding author
- Kumar, Pranjal
- ORCID
- Kumar, Pranjal
- Venue
- Artificial Intelligence Review
- Published
- 2024-08-18
- Volume
- 57
- Issue
- 10
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4401671778
- Cited by
- 395
- References
- 189
- Citations by year
- 2026: 219 · 2025: 162 · 2024: 14
- FWCI
- 58.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Artificial Intelligence Review (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, word embeddings, text generation, vision-language models, personalized learning, biomedical applications, code generation, GPT-4, deep learning, LLM limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large language models (LLMs): survey, technical frameworks, and future challenges. https://app.cochat.ai/papers/10.1007/s10462-024-10888-y
- The paper
- Kumar, P. (2024). Large language models (LLMs): survey, technical frameworks, and future challenges. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-10888-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.