CoChat

A Comprehensive Survey on Word Representation Models: From Classical to State-of-the-Art Word Representation Language Models

Open access
Naseem, Usman, Razzak, Imran, Khan, Shah Khalid, Prasad, Mukesh · 2021 · ACM Transactions on Asian and Low-Resource Language Information Processing

The DOI matches its Crossref record; 174 of the 174 references it cites were found in a registry or an index.

Abstract

Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP-related tasks. In the end, this survey briefly discusses the commonly used ML- and DL-based classifiers, evaluation metrics, and the applications of these word embeddings in different NLP tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3434237

Cites

174 of 174 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Attention Is All You Need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×

    “• Transformer-based Pre-trained Language Models Transformer [162] has been proven to be more efficient and faster than LSTM or CNN for language modelling, and thus the following advances in this domain will rely on this architecture.”
    BackgroundMethod
  • Long Short-Term Memory

    Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×

    “• Long Short-TermMemory (LSTM): LSTM was presented by Hochreiter and Schmidhuber [51].”
    Background
  • Deep learning

    LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×

    “directional Encoder Representations from Transformers (BERT) is a direct descendant of GPT: train a huge LM on free text and then fine-tune individual tasks without custom network architectures. BERT [30] is another contextualised word representation LM, where the transformer NN uses parallel attention layers rather t…”
    BackgroundInfluential
  • Gradient-based learning applied to document recognition

    Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×

    “y used for hierarchical classification in a DL [57]. CNN was built and used for image classification in early days, but over the period, it has shown excellent results for text classification as well [77]. In image classification, an image tensor is convolved with a set of kernels of size �x�. The convolution layers i…”
    Method
  • Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

    Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545

  • Distributed Representations of Words and Phrases and their Compositionality

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 4×

    “Continuous word representation models like Word2Vec , GloVe and FastText [16, 63, 89, 96] etc.”
    BackgroundMethodInfluential
  • Induction of decision trees

    Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×

    “• Decision Tree (DT) classifier: Decision tree (DT) was presented by Magerman [87] and developed by Quinlan [131].”
    Method
  • RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×

    “• RoBERTa: RoBERTa: A Robustly Optimized BERT Pre-training Approach was implemented in July 2019 [86]; it is like a light version of BERT, but it has fewer parameters and better performance as it removes the training on the sentence classification task.”
    Method
  • Convolutional Neural Networks for Sentence Classification

    Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · cited 1×

    “Qin et al. [128] used pre-trained Word2Vec embeddings and forwarded these word embeddings to CNN. Similarly, Kim [66] utilized pre-trained embeddings of Word2Vec and forwarded to CNN neural network, which increased the classification results.”
    Method

Show all 174 references

Research works printed but not linked: 28

  1. 10.5555/2535015 · doi:10.5555/2535015
  2. 10.5555/3235571 · doi:10.5555/3235571
  3. 10.5555/3157382.3157584 · doi:10.5555/3157382.3157584
  4. Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D . Manning . 2020 . Electra : Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). · arXiv
  5. 10.5555/3454287.3455457 · doi:10.5555/3454287.3455457
  6. 10.5555/2002472.2002489 · doi:10.5555/2002472.2002489
  7. 10.5555/2891460.2891697 · doi:10.5555/2891460.2891697
  8. 10.5555/1753126.1753129 · doi:10.5555/1753126.1753129
  9. 10.5555/3044805.3045025 · doi:10.5555/3044805.3045025
  10. 10.5555/2832415.2832428 · doi:10.5555/2832415.2832428
  11. 10.5555/559226 · doi:10.5555/559226
  12. 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
  13. 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
  14. 10.18653/v1/K16-1006 · DOI · Paper page
  15. 10.5555/2999792.2999959 · doi:10.5555/2999792.2999959
  16. 10.5555/3042817.3043083 · doi:10.5555/3042817.3043083
  17. 10.1016/S1002-0160(15)60047-9 · DOI
  18. Matthew E. Peters , Mark Neumann , Mohit Iyyer , Matt Gardner , Christopher Clark , Kenton Lee , and Luke Zettlemoyer . 2018. Deep contextualized word representations. CoRR abs/1802.05365 ( 2018 ). Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR abs/1802.05365 (2018). · arXiv
  19. 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
  20. Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2019 . Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
  21. 10.5555/3015812.3015844 · doi:10.5555/3015812.3015844
  22. 10.5555/1886436.1886447 · doi:10.5555/1886436.1886447
  23. 10.5555/106765.106782 · doi:10.5555/106765.106782
  24. 10.5555/3104482.3104610 · doi:10.5555/3104482.3104610
  25. 10.1109/TKDE.2015.2489653 · DOI · Paper page
  26. 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
  27. 10.5555/3297863.3297977 · doi:10.5555/3297863.3297977
  28. 10.5555/3454287.3454804 · doi:10.5555/3454287.3454804

Cited by

Items 101–195 of 195

Metadata

Authors
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M.
Affiliations
  • The University of Sydney (AU)
  • Deakin University (AU)
  • RMIT University (AU)
  • University of Technology Sydney (AU)
Venue
ACM Transactions on Asian and Low-Resource Language Information Processing
Published
2021-06-30
Volume
20
Issue
5
Pages
1-35
ISSN
  • 2375-4699 (ISSN-L)
  • 2375-4702
OpenAlex
W3095760691
Cited by
194
References
168
Citations by year
2026: 22 · 2025: 31 · 2024: 37 · 2023: 61 · 2022: 32 · 2021: 10
FWCI
12.26
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Asian and Low-Resource Language Information Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • ResearchOnline at James Cook University (James Cook University) (submittedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RMIT Research Repository (RMIT University Library) (submittedVersion)
  • Deakin Research Online (Deakin University) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Figshare (submittedVersion)
  • OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
word representation, word embeddings, text representation, evaluation metrics, low-resource languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.