A Comprehensive Survey on Word Representation Models: From Classical to State-of-the-Art Word Representation Language Models
Open accessThe DOI matches its Crossref record; 174 of the 174 references it cites were found in a registry or an index.
Abstract
Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP-related tasks. In the end, this survey briefly discusses the commonly used ML- and DL-based classifiers, evaluation metrics, and the applications of these word embeddings in different NLP tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Doc2vec can be a powerful feature extractor but fails to capture out-of-vocabulary words (Naseem et al., 2021).”
From Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders (2023) · cites this paper as Naseem et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“• Transformer-based Pre-trained Language Models Transformer [162] has been proven to be more efficient and faster than LSTM or CNN for language modelling, and thus the following advances in this domain will rely on this architecture.”
BackgroundMethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“• Long Short-TermMemory (LSTM): LSTM was presented by Hochreiter and Schmidhuber [51].”
BackgroundLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“directional Encoder Representations from Transformers (BERT) is a direct descendant of GPT: train a huge LM on free text and then fine-tune individual tasks without custom network architectures. BERT [30] is another contextualised word representation LM, where the transformer NN uses parallel attention layers rather t…”
BackgroundInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“y used for hierarchical classification in a DL [57]. CNN was built and used for image classification in early days, but over the period, it has shown excellent results for text classification as well [77]. In image classification, an image tensor is convolved with a set of kernels of size �x�. The convolution layers i…”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 4×
“Continuous word representation models like Word2Vec , GloVe and FastText [16, 63, 89, 96] etc.”
BackgroundMethodInfluentialQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“• Decision Tree (DT) classifier: Decision tree (DT) was presented by Magerman [87] and developed by Quinlan [131].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“• RoBERTa: RoBERTa: A Robustly Optimized BERT Pre-training Approach was implemented in July 2019 [86]; it is like a light version of BERT, but it has fewer parameters and better performance as it removes the training on the sentence classification task.”
MethodConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · cited 1×
“Qin et al. [128] used pre-trained Word2Vec embeddings and forwarded these word embeddings to CNN. Similarly, Kim [66] utilized pre-trained embeddings of Word2Vec and forwarded to CNN neural network, which increased the classification results.”
Method
Research works printed but not linked: 28
- 10.5555/2535015 · doi:10.5555/2535015
- 10.5555/3235571 · doi:10.5555/3235571
- 10.5555/3157382.3157584 · doi:10.5555/3157382.3157584
- Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D . Manning . 2020 . Electra : Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). · arXiv
- 10.5555/3454287.3455457 · doi:10.5555/3454287.3455457
- 10.5555/2002472.2002489 · doi:10.5555/2002472.2002489
- 10.5555/2891460.2891697 · doi:10.5555/2891460.2891697
- 10.5555/1753126.1753129 · doi:10.5555/1753126.1753129
- 10.5555/3044805.3045025 · doi:10.5555/3044805.3045025
- 10.5555/2832415.2832428 · doi:10.5555/2832415.2832428
- 10.5555/559226 · doi:10.5555/559226
- 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
- 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
- 10.18653/v1/K16-1006 · DOI · Paper page
- 10.5555/2999792.2999959 · doi:10.5555/2999792.2999959
- 10.5555/3042817.3043083 · doi:10.5555/3042817.3043083
- 10.1016/S1002-0160(15)60047-9 · DOI
- Matthew E. Peters , Mark Neumann , Mohit Iyyer , Matt Gardner , Christopher Clark , Kenton Lee , and Luke Zettlemoyer . 2018. Deep contextualized word representations. CoRR abs/1802.05365 ( 2018 ). Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR abs/1802.05365 (2018). · arXiv
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2019 . Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
- 10.5555/3015812.3015844 · doi:10.5555/3015812.3015844
- 10.5555/1886436.1886447 · doi:10.5555/1886436.1886447
- 10.5555/106765.106782 · doi:10.5555/106765.106782
- 10.5555/3104482.3104610 · doi:10.5555/3104482.3104610
- 10.1109/TKDE.2015.2489653 · DOI · Paper page
- 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
- 10.5555/3297863.3297977 · doi:10.5555/3297863.3297977
- 10.5555/3454287.3454804 · doi:10.5555/3454287.3454804
Cited by
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
“We refer the readers interested in smaller models to [35], [36], [32].”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“We refer the readers interested in smaller models to [35], [36], [32].”
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Information · doi:10.3390/info15090517 · Cited by 732
COVIDSenti: A Large-Scale Benchmark Twitter Data Set for COVID-19 Sentiment Analysis
Naseem, U., Razzak, I., Khushi, M., Eklund, P. W., & Kim, J. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3051189 · Cited by 343
“In addition, we used hybrid models, such as hybrid ranking (HyRank) and Improved Word Vector (IWV), that incorporate sentiment and context of tweets for Twitter sentiment analysis [36].”
MethodSelf-citationEssel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100198 · Cited by 296
“ChatGPT is trained to estimate the probability of a specific succession of words founded on the context of the words that preceded it (Liu et al., 2023; Naseem et al., 2021).”
Background
Metadata
- Authors
- Naseem, U., Razzak, I., Khan, S. K., & Prasad, M.
- Affiliations
- The University of Sydney (AU)
- Deakin University (AU)
- RMIT University (AU)
- University of Technology Sydney (AU)
- Venue
- ACM Transactions on Asian and Low-Resource Language Information Processing
- Published
- 2021-06-30
- Volume
- 20
- Issue
- 5
- Pages
- 1-35
- ISSN
- 2375-4699 (ISSN-L)
- 2375-4702
- OpenAlex
- W3095760691
- Cited by
- 194
- References
- 168
- Citations by year
- 2026: 22 · 2025: 31 · 2024: 37 · 2023: 61 · 2022: 32 · 2021: 10
- FWCI
- 12.26
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Asian and Low-Resource Language Information Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- ResearchOnline at James Cook University (James Cook University) (submittedVersion)
- arXiv (Cornell University) (submittedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Deakin Research Online (Deakin University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Figshare (submittedVersion)
- OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word representation, word embeddings, text representation, evaluation metrics, low-resource languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Comprehensive Survey on Word Representation Models: From Classical to State-of-the-Art Word Representation Language Models. https://app.cochat.ai/papers/10.1145/3434237
- The paper
- Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021). A Comprehensive Survey on Word Representation Models: From Classical to State-of-the-Art Word Representation Language Models. ACM Transactions on Asian and Low-Resource Language Information Processing. https://doi.org/10.1145/3434237
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.