CoChat

A Comprehensive Survey on Word Representation Models: From Classical to State-of-the-Art Word Representation Language Models

Open access
Naseem, Usman, Razzak, Imran, Khan, Shah Khalid, Prasad, Mukesh · 2021 · ACM Transactions on Asian and Low-Resource Language Information Processing

The DOI matches its Crossref record; 174 of the 174 references it cites were found in a registry or an index.

Abstract

Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP-related tasks. In the end, this survey briefly discusses the commonly used ML- and DL-based classifiers, evaluation metrics, and the applications of these word embeddings in different NLP tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3434237

Cites

174 of 174 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 28

  1. 10.5555/2535015 · doi:10.5555/2535015
  2. 10.5555/3235571 · doi:10.5555/3235571
  3. 10.5555/3157382.3157584 · doi:10.5555/3157382.3157584
  4. Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D . Manning . 2020 . Electra : Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). · arXiv
  5. 10.5555/3454287.3455457 · doi:10.5555/3454287.3455457
  6. 10.5555/2002472.2002489 · doi:10.5555/2002472.2002489
  7. 10.5555/2891460.2891697 · doi:10.5555/2891460.2891697
  8. 10.5555/1753126.1753129 · doi:10.5555/1753126.1753129
  9. 10.5555/3044805.3045025 · doi:10.5555/3044805.3045025
  10. 10.5555/2832415.2832428 · doi:10.5555/2832415.2832428
  11. 10.5555/559226 · doi:10.5555/559226
  12. 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
  13. 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
  14. 10.18653/v1/K16-1006 · DOI · Paper page
  15. 10.5555/2999792.2999959 · doi:10.5555/2999792.2999959
  16. 10.5555/3042817.3043083 · doi:10.5555/3042817.3043083
  17. 10.1016/S1002-0160(15)60047-9 · DOI
  18. Matthew E. Peters , Mark Neumann , Mohit Iyyer , Matt Gardner , Christopher Clark , Kenton Lee , and Luke Zettlemoyer . 2018. Deep contextualized word representations. CoRR abs/1802.05365 ( 2018 ). Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR abs/1802.05365 (2018). · arXiv
  19. 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
  20. Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2019 . Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
  21. 10.5555/3015812.3015844 · doi:10.5555/3015812.3015844
  22. 10.5555/1886436.1886447 · doi:10.5555/1886436.1886447
  23. 10.5555/106765.106782 · doi:10.5555/106765.106782
  24. 10.5555/3104482.3104610 · doi:10.5555/3104482.3104610
  25. 10.1109/TKDE.2015.2489653 · DOI · Paper page
  26. 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
  27. 10.5555/3297863.3297977 · doi:10.5555/3297863.3297977
  28. 10.5555/3454287.3454804 · doi:10.5555/3454287.3454804

Cited by

See all 195

Metadata

Authors
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M.
Affiliations
  • The University of Sydney (AU)
  • Deakin University (AU)
  • RMIT University (AU)
  • University of Technology Sydney (AU)
Venue
ACM Transactions on Asian and Low-Resource Language Information Processing
Published
2021-06-30
Volume
20
Issue
5
Pages
1-35
ISSN
  • 2375-4699 (ISSN-L)
  • 2375-4702
OpenAlex
W3095760691
Cited by
194
References
168
Citations by year
2026: 22 · 2025: 31 · 2024: 37 · 2023: 61 · 2022: 32 · 2021: 10
FWCI
12.26
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Asian and Low-Resource Language Information Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • ResearchOnline at James Cook University (James Cook University) (submittedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RMIT Research Repository (RMIT University Library) (submittedVersion)
  • Deakin Research Online (Deakin University) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Figshare (submittedVersion)
  • OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
word representation, word embeddings, text representation, evaluation metrics, low-resource languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.