A Comprehensive Survey on Word Representation Models: From Classical to State-of-the-Art Word Representation Language Models
Open accessThe DOI matches its Crossref record; 174 of the 174 references it cites were found in a registry or an index.
Abstract
Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP-related tasks. In the end, this survey briefly discusses the commonly used ML- and DL-based classifiers, evaluation metrics, and the applications of these word embeddings in different NLP tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Doc2vec can be a powerful feature extractor but fails to capture out-of-vocabulary words (Naseem et al., 2021).”
From Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders (2023) · cites this paper as Naseem et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“• Transformer-based Pre-trained Language Models Transformer [162] has been proven to be more efficient and faster than LSTM or CNN for language modelling, and thus the following advances in this domain will rely on this architecture.”
BackgroundMethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“• Long Short-TermMemory (LSTM): LSTM was presented by Hochreiter and Schmidhuber [51].”
BackgroundLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“directional Encoder Representations from Transformers (BERT) is a direct descendant of GPT: train a huge LM on free text and then fine-tune individual tasks without custom network architectures. BERT [30] is another contextualised word representation LM, where the transformer NN uses parallel attention layers rather t…”
BackgroundInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“y used for hierarchical classification in a DL [57]. CNN was built and used for image classification in early days, but over the period, it has shown excellent results for text classification as well [77]. In image classification, an image tensor is convolved with a set of kernels of size �x�. The convolution layers i…”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 4×
“Continuous word representation models like Word2Vec , GloVe and FastText [16, 63, 89, 96] etc.”
BackgroundMethodInfluentialQuinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · cited 1×
“• Decision Tree (DT) classifier: Decision tree (DT) was presented by Magerman [87] and developed by Quinlan [131].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“• RoBERTa: RoBERTa: A Robustly Optimized BERT Pre-training Approach was implemented in July 2019 [86]; it is like a light version of BERT, but it has fewer parameters and better performance as it removes the training on the sentence classification task.”
MethodConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · cited 1×
“Qin et al. [128] used pre-trained Word2Vec embeddings and forwarded these word embeddings to CNN. Similarly, Kim [66] utilized pre-trained embeddings of Word2Vec and forwarded to CNN neural network, which increased the classification results.”
Method
Research works printed but not linked: 28
- 10.5555/2535015 · doi:10.5555/2535015
- 10.5555/3235571 · doi:10.5555/3235571
- 10.5555/3157382.3157584 · doi:10.5555/3157382.3157584
- Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D . Manning . 2020 . Electra : Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). · arXiv
- 10.5555/3454287.3455457 · doi:10.5555/3454287.3455457
- 10.5555/2002472.2002489 · doi:10.5555/2002472.2002489
- 10.5555/2891460.2891697 · doi:10.5555/2891460.2891697
- 10.5555/1753126.1753129 · doi:10.5555/1753126.1753129
- 10.5555/3044805.3045025 · doi:10.5555/3044805.3045025
- 10.5555/2832415.2832428 · doi:10.5555/2832415.2832428
- 10.5555/559226 · doi:10.5555/559226
- 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
- 10.5555/3295222.3295377 · doi:10.5555/3295222.3295377
- 10.18653/v1/K16-1006 · DOI · Paper page
- 10.5555/2999792.2999959 · doi:10.5555/2999792.2999959
- 10.5555/3042817.3043083 · doi:10.5555/3042817.3043083
- 10.1016/S1002-0160(15)60047-9 · DOI
- Matthew E. Peters , Mark Neumann , Mohit Iyyer , Matt Gardner , Christopher Clark , Kenton Lee , and Luke Zettlemoyer . 2018. Deep contextualized word representations. CoRR abs/1802.05365 ( 2018 ). Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR abs/1802.05365 (2018). · arXiv
- 10.1023/A:1022643204877 · doi:10.1023/a:1022643204877 · Paper page
- Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2019 . Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
- 10.5555/3015812.3015844 · doi:10.5555/3015812.3015844
- 10.5555/1886436.1886447 · doi:10.5555/1886436.1886447
- 10.5555/106765.106782 · doi:10.5555/106765.106782
- 10.5555/3104482.3104610 · doi:10.5555/3104482.3104610
- 10.1109/TKDE.2015.2489653 · DOI · Paper page
- 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
- 10.5555/3297863.3297977 · doi:10.5555/3297863.3297977
- 10.5555/3454287.3454804 · doi:10.5555/3454287.3454804
Cited by
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
“We refer the readers interested in smaller models to [35], [36], [32].”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“We refer the readers interested in smaller models to [35], [36], [32].”
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Information · doi:10.3390/info15090517 · Cited by 732
COVIDSenti: A Large-Scale Benchmark Twitter Data Set for COVID-19 Sentiment Analysis
Naseem, U., Razzak, I., Khushi, M., Eklund, P. W., & Kim, J. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3051189 · Cited by 343
“In addition, we used hybrid models, such as hybrid ranking (HyRank) and Improved Word Vector (IWV), that incorporate sentiment and context of tweets for Twitter sentiment analysis [36].”
MethodSelf-citationEssel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100198 · Cited by 296
“ChatGPT is trained to estimate the probability of a specific succession of words founded on the context of the words that preceded it (Liu et al., 2023; Naseem et al., 2021).”
BackgroundTransformer-based deep learning models for the sentiment analysis of social media data
Tabinda Kokab, S., Asghar, S., & Naz, S. (2022) · Array · doi:10.1016/j.array.2022.100157 · Cited by 237
Investigating the Use of ChatGPT for the Scheduling of Construction Projects
Prieto, S. A., Mengiste, E. T., & García de Soto, B. (2023) · Buildings · doi:10.3390/buildings13040857 · Cited by 229
“Within the field of NLP, research has been focused on developing Large Language representation Models (LLM) that expand the applicability of NLP to more detailed human language understanding tasks such as translation, text classification, and holding conversations [11,12].”
MethodFinancial Sentiment Analysis: Techniques and Applications
Du, K., Xing, F., Mao, R., & Cambria, E. (2024) · ACM Computing Surveys · doi:10.1145/3649451 · Cited by 173
Industrial applications of large language models
Raza, M., Jahangir, Z., Riaz, M. B., Saeed, M. J., & Sattar, M. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-98483-1 · Cited by 163
Islam, M. S., Kabir, M. N., Ghani, N. A., Zamli, K. Z., Zulkifli, N. S. A., Rahman, M. M., & Moni, M. A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10651-9 · Cited by 227
Just, J. (2023) · Technovation · doi:10.1016/j.technovation.2023.102883 · Cited by 145
Sunitha, D., Patra, R. K., Babu, N. V., Suresh, A., & Gupta, S. C. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.04.027 · Cited by 117
Bereczki, Z., Benczik, B., Balogh, O. M., Marton, S., Puhl, E., Pétervári, M., Váczy‐Földi, M., Papp, Z. T., Makkos, A., Glass, K., et al. (2024) · British Journal of Pharmacology · doi:10.1111/bph.17302 · Cited by 94
“Consequently, these embedding algorithms no longer learn to reconstruct the input vector but rather to accurately predict the relationship between pairs of input vectors (Bengio et al., 2003; Naseem et al., 2021).”
Hybrid ensemble framework with self-attention mechanism for social spam detection on imbalanced data
Rao, S., Verma, A. K., & Bhatia, T. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119594 · Cited by 74
Vision-Language Transformer for Interpretable Pathology Visual Question Answering
Naseem, U., Khushi, M., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3163751 · Cited by 63
“…[3] and models such as Bidirectional Encoder Representations from Transform-ers (BERT) [4], Generative Pre-trained Transformer (GPT) [5] and others [6]–[8] the success of NLP and the strong representation capabilities of transformers, it has attracted tremendous interest and proved its dominance…”
MethodSelf-citationHate Speech Detection Using Large Language Models: A Comprehensive Review
Albladi, A., Islam, M., Das, A., Bigonah, M., Zhang, Z., Jamshidi, F., Rahgouy, M., Raychawdhary, N., Marghitu, D., & Seals, C. (2025) · IEEE Access · doi:10.1109/access.2025.3532397 · Cited by 127
“A significant breakthrough came in 2013 with the introduction of the word2vec model, which utilized neural networks to create dense vector representations of words [47], [48].”
A Comparative Analysis of Active Learning for Biomedical Text Mining
Naseem, U., Khushi, M., Khan, S. K., Shaukat, K., & Moni, M. A. (2021) · Applied System Innovation · doi:10.3390/asi4010023 · Cited by 82
“AL techniques use supervised ML methods in an iterative way.”
MethodSelf-citationLinguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Swiatczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 54
“Given the success of high-capacity natural language LMs without linguistic priors on many NLP tasks 27–30 , it might be tempting to also build protein LMs with minimal biological priors.”
BackgroundMethodTransformer-Based Graph Convolutional Network for Sentiment Analysis
AlBadani, B., Shi, R., Dong, J., Al-sabri, R., & Moctard, O. B. (2022) · Applied Sciences · doi:10.3390/app12031316 · Cited by 45
“Of the various machine learning classification methods used to classify users’ sentiments from a text, decision tree, LDA, Naive Bayes, Support Vector Machine (SVM), and artificial neural networks are the most common and have achieved a higher performance [9,22,27,28].”
BackgroundMethodJing, X., Wu, Z., Zhang, L., Li, Z., & Mu, D. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2023.3269463 · Cited by 52
“Thusfar,tousetheWord2Vecusually coupling with CBOW becomes a standard setting for developing pretrained word representation [19].”
Recommender Systems Based on Graph Embedding Techniques: A Review
Deng, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3174197 · Cited by 60
“Techniques based on meta path Meta path-based techniques were inspired from the idea of word representation [257] in natural language processing (NLP): given N words chosen from a corpus containing hundreds of millions of words; then use them to try to make up a legal sentence like w1, w2, .”
MethodStudy of Word Embeddings for Enhanced Cyber Security Named Entity Recognition
Srivastava, S., Paul, B., & Gupta, D. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.027 · Cited by 60
Tawil, A. A., Almazaydeh, L. H., Qawasmeh, D., Qawasmeh, B., Alshinwan, M., & Elleithy, K. M. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.057279 · Cited by 83
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., et al. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1178181 · Cited by 36
“These techniques apply many different forms of processing, each having its advantages and disadvantages (Naseem et al., 2021).”
BackgroundThe fusion of fuzzy theories and natural language processing: A state-of-the-art survey
Liu, M., Zhang, H., Xu, Z., & Ding, K. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111818 · Cited by 47
Gani, M. O., Ayyasamy, R. K., Sangodiah, A., & Fui, Y. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11842-1 · Cited by 50
“It has fewer parameters than the original BERT and removed the next sentence prediction task (Naseem et al., 2021).”
BackgroundFault Detection in Wireless Sensor Network Based on Deep Learning Algorithms
Regin, R., Rajest, S. S., & Singh, B. (2018) · ICST Transactions on Scalable Information Systems · doi:10.4108/eai.3-5-2021.169578 · Cited by 32
“This methodology can handle huge information sizes but can be applied only if a central processor is accessible to join the incomplete, partial support vectors [17-24].”
BackgroundTowards Practical Secure Neural Network Inference: The Journey So Far and the Road Ahead
Mann, Z. Á., Weinert, C., Chabal, D., & Bos, J. W. (2023) · ACM Computing Surveys · doi:10.1145/3628446 · Cited by 48
“Readers aiming at understanding the details of the relevant secure computation techniques will need advanced cryptographic knowledge.”
BackgroundNaseem, U., Lee, B. C., Khushi, M., Kim, J., & Dunn, A. G. (2022) · doi:10.18653/v1/2022.nlppower-1.3 · Cited by 44
Self-citationHybrid Text Representation for Explainable Suicide Risk Identification on Social Media
Naseem, U., Khushi, M., Kim, J., & Dunn, A. G. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3184984 · Cited by 43
“Deep learning (DL) has resulted in a substantial performance boost over the state of the art (SOTA) for various NLP applications [39], including suicide ideation on social media.”
BackgroundSelf-citationAI-based novelty detection in crowdsourced idea spaces
Just, J., Ströhle, T., Füller, J., & Hutter, K. (2023) · Innovation · doi:10.1080/14479338.2023.2215740 · Cited by 40
“In the last few years, the development of language models for semantic text representation advanced significantly (Naseem et al., 2021).”
K-PathVQA: Knowledge-Aware Multimodal Representation for Pathology Visual Question Answering
Naseem, U., Khushi, M., Dunn, A. G., & Kim, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3294249 · Cited by 40
“Theyhavebeenshownto achieveSOTAperformanceinvariousNLPtasks,includingtext classificationandlanguagegeneration[34].”
Self-citationIdentification of bacteriophage genome sequences with representation learning
Bai, Z., Zhang, Y., Miyano, S., Yamaguchi, R., Fujimoto, K., Uematsu, S., & Imoto, S. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac509 · Cited by 29
“Here, we used Masked Language Modeling (Devlin et al. , 2018; Naseem et al. , 2021) as the pre-training task, which is the same as Ji et al. (2021).”
MethodFusing external knowledge resources for natural language understanding techniques: A survey
Wang, Y., Wang, W., Chen, Q., Huang, K., Nguyen, A., De, S., & Hussain, A. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.025 · Cited by 32
Combining Graph Neural Networks and Sentence Encoders for Knowledge-aware Recommendations
Spillo, G., Musto, C., Polignano, M., Lops, P., de Gemmis, M., & Semeraro, G. M. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3565472.3592965 · Cited by 32
“Despite this, recent benchmarks showed that models based on transformers currently represent the state-ofthe-art in the area [40].”
BackgroundA systematic review of AI-driven business models for advancing Sustainable Development Goals
Khan, M. A., Rahman, A., Mahmud, F. U., Bishnu, K. K., Ahmed, M., Mridha, M. F., & Aung, Z. (2025) · Array · doi:10.1016/j.array.2025.100539 · Cited by 34
Character gated recurrent neural networks for Arabic sentiment analysis
Omara, E., Mousa, M., & Ismail, N. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-13153-w · Cited by 26
“Neural networks are commonly used for learning distributed representation of text, known as word embedding 27,29 .”
BackgroundMethodInfluentialSystematic literature review of arabic aspect-based sentiment analysis
Alyami, S., Alhothali, A., & Jamal, A. T. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.07.001 · Cited by 34
“In contrast to manually created features, a distributed word representation (word embedding) is built automatically using unsupervised learning techniques (Naseem et al., 2021).”
Evaluating Intelligent Methods for Detecting COVID-19 Fake News on Social Media Platforms
Alhakami, H., Alhakami, W., Baz, A., Faizan, M., Khan, M. W., & Agrawal, A. (2022) · Electronics · doi:10.3390/electronics11152417 · Cited by 23
“There are many word embedding techniques such as the Continuous Bag of Words, Skip-Gram, GloVe etc. [38].”
MethodUse of Social Media Data to Diagnose and Monitor Psychotic Disorders: Systematic Review
Lejeune, A., Robaglia, B.-M., Walter, M., Berrouiguet, S., & Lemey, C. (2022) · Journal of Medical Internet Research · doi:10.2196/36986 · Cited by 21
“Thus, these feature extraction methods fall short when capturing semantic or syntactic information or the sentiments of words [42].”
Natural language processing for social science research: A comprehensive review
Hou, Y., & Huang, J. (2025) · Chinese Journal of Sociology · doi:10.1177/2057150x241306780 · Cited by 45
“Text preprocessing enhances data quality and feature extraction (Naseem et al., 2021), in which researchers reduce the noise in data by eliminating irrelevant information, emojis, spelling errors, etc.”
Language Representation Models: An Overview
Schomacker, T., & Tropmann-Frick, M. (2021) · Entropy · doi:10.3390/e23111422 · Cited by 22
“The authors of [5] conducted a rich survey of language representation models, including many models from 2020 and 2021.”
BackgroundCan large language models automate the HAZOP process without human intervention?
Lee, J., Park, S., Oh, S., & Ma, B.-C. (2025) · Safety Science · doi:10.1016/j.ssci.2025.107039 · Cited by 23
A Novel Approach for Implementing Conventional LBIST by High Execution Microprocessors
Shravani, C., Krishna, G. R., Bollam, H. L., Vatambeti, R., & Saikumar, K. (2022) · International Conference on Smart Systems and Inventive Technology (ICSSIT) · doi:10.1109/icssit53264.2022.9716411 · Cited by 17
Combs, K., Lu, H., & Bihl, T. (2023) · Algorithms · doi:10.3390/a16030146 · Cited by 24
Cross-lingual Sentiment Analysis of Tamil Language Using a Multi-stage Deep Learning Architecture
Prakash, J., V, & Vijay, S. A. A. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3631391 · Cited by 18
“Tamil News Classification dataset [33] consists of news articles in the Tamil language, with six categories: “tamilnadu,” “india,” “cinema,” “sports,” “politics,” and “world.”
Clinical Text Classification with Word Representation Features and Machine Learning Algorithms
Almazaydeh, L. H., Abuhelaleh, M. A., Tawil, A. A., & Elleithy, K. M. (2023) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v19i04.36099 · Cited by 26
Pradhan, R., & Sharma, D. K. (2022) · Soft Computing · doi:10.1007/s00500-022-07091-y · Cited by 18
“Pretrained models are classified into casual language models and masked language models (Naseem et al. 2021).”
BackgroundRetractedTeijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Siso, P. V., van den Brand, S. A. G. E., Bockting, C., van de Schoot, R., et al. (2022) · doi:10.31234/osf.io/t7bpd · Cited by 14
Multimodal consistency-specificity fusion based on information bottleneck for sentiment analysis
Liu, W., Cao, S., & Zhang, S. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.101943 · Cited by 19
“Similar to the intuition of unsupervised pre-trained language models (Naseem et al., 2021), the co-occurrence of heterogeneous modalities is a type of potential self-supervised signal, which also reveals the existence of consistent or modality-correlated information.”
TunBERT: Pretraining BERT for Tunisian Dialect Understanding
Haddad, H., Rouhou, A. C., Messaoudi, A., Korched, A., Fourati, C., Sellami, A., HajHmida, M. B., & Ghriss, F. (2023) · SN Computer Science · doi:10.1007/s42979-022-01541-y · Cited by 13
Tang, V., & Paìnho, M. (2023) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2023.2213869 · Cited by 16
“Word embeddings are vector representations of text data that enable semantic properties to be encoded whereby similar pieces of text information are nearer in vector space (Naseem et al. 2021).”
BackgroundDeep Learning in NLP for Anomalous HTTP Requests Detection
Anh Nguyen, M. T., Tong, V., Souihi, S. B., & Souihi, S. (2023) · 2023 19th International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm59352.2023.10327888 · Cited by 12
“Word embedding techniques are among the most efficient NLP methods [10].”
MethodMachine Learning Algorithms for Predicting SARS-CoV-2 (COVID-19) – A Comparative Analysis
Mary, L. W., & Raj, S. A. A. (2021) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec51865.2021.9591801 · Cited by 12
“It's the most common method for categorizing text documents and categorical variables [24-25].”
MethodMulti-Aspect Annotation and Analysis of Nepali Tweets on Anti-Establishment Election Discourse
Rauniyar, K., Poudel, S., Shiwakoti, S., Thapa, S., Rashid, J., Kim, J., Imran, M. A., & Naseem, U. (2023) · IEEE Access · doi:10.1109/access.2023.3342154 · Cited by 27
“New possibilities for data analysis and information extraction have been made possible by ground-breaking developments in NLP, which has enabled machines to interpret human language with very high precision [25].”
BackgroundSelf-citationMutemi, A., & Bação, F. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100256 · Cited by 21
Guided Semi-Supervised Non-Negative Matrix Factorization
Li, P., Tseng, C., Zheng, Y., Chew, J. A., Huang, L., Jarman, B., & Needell, D. M. (2022) · Algorithms · doi:10.3390/a15050136 · Cited by 12
Jovic, M., Papakonstantinidis, S., & Kirkpatrick, R. (2025) · Language Testing in Asia · doi:10.1186/s40468-025-00389-2 · Cited by 11
“Naseem et al. (2021) define prompt engineering as the systematic crafting of effective prompts to enhance AI responses, while Tirumala (2022) highlights its essential role in improving feedback quality and reliability.”
Naseem, U., Khushi, M., Kim, J., & Dunn, A. G. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533454 · Cited by 13
Self-citationCheng, X., Mei, T., Zi, Y., Wang, Q., Gao, Z., & Yang, H. (2024) · IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS) · doi:10.1109/icpics62053.2024.10796626 · Cited by 14
Zero Trust: Deep Learning and NLP for HTTP Anomaly Detection in IDS
Nguyen, M. T. A., Tong, V., Bannour, S., & Souihi, S. (2025) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2025.3560040 · Cited by 13
Graph-Based Hierarchical Attention Network for Suicide Risk Detection on Social Media
Naseem, U., Kim, J., Khushi, M., & Dunn, A. G. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587587 · Cited by 12
Self-citationDiviya, M., & Karmel, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-41484-9 · Cited by 13
Agüero-Torales, M. M., López-Herrera, A. G., & Vilares, D. (2023) · Cognitive Computation · doi:10.1007/s12559-023-10165-0 · Cited by 17
“We focus on masked language models since they are the most popular choice as a backbone for downstream natural language processing and language and vision models [83, 84].”
BackgroundActive Learning with an Adaptive Classifier for Inaccessible Big Data Analysis
Jahan, S., Islam, M. R., Hasib, K. M., Naseem, U., & Islam, M. S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534046 · Cited by 9
“This means that the class provides limited specificity’s for the minority class, while the class offers great precision [9], [10].”
BackgroundSelf-citationLin, S.-J., Yeh, W.-C., Chiu, Y.-W., Chang, Y., Hsu, M., Chen, Y.-S., & Hsu, W. (2022) · Database · doi:10.1093/database/baac056 · Cited by 11
A Transformer based approach for Abstractive Text Summarization of Radiology Reports
Balouch, B. A. K., & Hussain, F. (2023) · International Conference on Applied Engineering and Natural Sciences · doi:10.59287/icaens.1042 · Cited by 10
“In contrast, contextual embedding models, such as ELMo (Embeddings from Language Models) and BERT (Bidirectional Encoder Representations from Transformers), generate dynamic embedding that capture the meaning of a word based on its context within a sentence [25].”
MethodAbdalsalam, M., Li, C., Dahou, A., & Kryvinska, N. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.029911 · Cited by 10
Methods for Acquiring and Incorporating Knowledge into Stock Price Prediction: A Survey
Wang, L., Li, J., Zhao, L., Kou, Z., Wang, X., Zhu, X., Wang, H., Shen, Y., & Chen, L. (2025) · ACM Computing Surveys · doi:10.1145/3773079 · Cited by 18
A Flat-Hierarchical Approach Based on Machine Learning Model for e-Commerce Product Classification
Mamani, H. E. C., Saboya, N., Rodrigues, P. C., Salas, R., & López‐Gonzales, J. L. (2024) · IEEE Access · doi:10.1109/access.2024.3400693 · Cited by 12
“The first one calculates the term frequency (TF) in a document, and the second technique combines the first one with the Inverse Document Frequency (TF-IDF) to reduce the impact of common words [50].”
BackgroundMethodInfluentialSarcasm Detection in Newspaper Headlines
Prasad, B. S., Babu, N. S., Reddy, T., Reddy, A. V. V., Vekkot, S., & Nair, P. C. (2024) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit61985.2024.10627150 · Cited by 7
“Word embeddings form the foundation of semantic representation, translating the complexities of language into mathematical vectors, providing a means to represent words in a format that is understandable to machine learning models [29].”
BackgroundShingjergji, K., Çelebi, R., Scholtes, J., & Dumontier, M. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103902 · Cited by 7
Wang, J., Li, H., Du, X., Hung, J.-L., & Yang, S. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101594 · Cited by 7
“With the success of pre-trained language models, recent studies have applied them (such as ELMo, GPT, Bert and XLNet) to a variety of downstream tasks in a light fine-tuning way.”
Slotting Learning Rate in Deep Neural Networks to Build Stronger Models
Sharma, D. K., Singh, B., Anam, M., Villalba‐Condori, K. O., Gupta, A., & Ali, G. K. (2021) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec51865.2021.9591733 · Cited by 6
Optimizing loss functions for improved energy demand prediction in smart power grids
Nusipova, F., Rysbekov, S., Абдиахметова, З., & Kartbayev, A. Z. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i3.pp3415-3426 · Cited by 12
“This approach is inspired by the word2vec word representation learning method [26], where the goal is to linearly separate pairs of context and word instances.”
MethodSolving Turkish math word problems by sequence-to-sequence encoder-decoder models
GEDİK, E., & Güngör, T. (2023) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3993 · Cited by 9
“These models capture the context of the words, the syntactic and semantic similarities of the words, and the relationship of the words to other words [28].”
BackgroundArabic text detection: a survey of recent progress challenges and opportunities
Muaad, A. Y. M., Raza, S., Naseem, U., & Davanagere, H. J. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04992-9 · Cited by 6
“There many works for other languages which address only one stage such as representation and-classification [21, 22].”
Self-citationWang, T., Long, T., Mei, A., Li, J., & Liu, Z. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.102025 · Cited by 7
Azeez, N. A., Misra, S., Ogaraku, D. O., & Abidoye, A. P. (2024) · Sensors · doi:10.3390/s24175817 · Cited by 11
“The text classification pipeline is provided in Figure 1 [31].”
MethodJilcha, L. A., & Kwak, J. (2021) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.023167 · Cited by 5
Video Summarization using Text Subjectivity Classification
Moraes, L. M. P., Marcacini, R. M., & Goularte, R. (2022) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3539637.3556998 · Cited by 8
“Pre-trained language models for text embeddings, such as BERT [12], was proven effective in many nat-ural language processing tasks [30].”
MethodMudgil, A., Rauniyar, K., Goel, R., Thapa, S., & Negi, A. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-99421-7.00006-4 · Cited by 8
Evaluating Contextualized Embeddings for Topic Modeling in Public Bidding Domain
Hott, H. R., Silva, M. O., Oliveira, G. P., Brandão, M. A., Lacerda, A. M., & Pappa, G. L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45392-2_27 · Cited by 7
LogCTBL: a hybrid deep learning model for log-based anomaly detection
Huang, H., Luo, W., Wang, Y., Zhou, Y., & Huang, W. T. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-06926-3 · Cited by 6
“Traditionally, Word-2Vec [25] or GloVe [26] are used to vectorize sentences.”
Retracted: Technical Support for Detection and Prediction of Rainfall
Hemavathi, D., Kumar, V. R., Regin, R., Rajest, S. S., Phasinam, K., & Singh, S. (2021) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec51865.2021.9591762 · Cited by 5
“…and statistical) approach for thunderstorm prediction and related lighting over the SHAR (ISRO rocket launch site), which ISRO uses to predict [4]. e) The function of monsoon surface processes using observations and regional climate models creates high-resolution ground surface data set for…”
MethodRetractedChatGPT and library users: AI risks of hallucinations and misinformation
Oladokun, B. D., Emmanuel, V. O., Osagie, O., & Alabi, A. O. (2025) · Cybrarians Journal · doi:10.70000/cj.2025.76.642 · Cited by 6
“The model's training process involves two main phases: pre-training, where the model learns to predict the next word in a sentence, and fine-tuning, where it is further refined using a narrower dataset with human feedback to enhance its accuracy and appropriateness (Naseem et al., 2021).”
Arabic Cyberbullying Detection Using Machine Learning: State of the Art Survey
Alsunaidi, N., Aljbali, S., Yasin, Y., & Aljamaan, H. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593968 · Cited by 4
“There are several models for word representation including Bag of Words (BoW), Word Embedding (WE), and Term Frequency-Inverse Document Frequency (TF-IDF) [18].”
MethodShah-Mohammadi, F., & Finkelstein, J. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385329 · Cited by 4
Ahmad, Z., Mohamed, A., Conway, M., & Maskat, R. (2024) · International Conference on Artificial Intelligence and Data Sciences (AiDAS) · doi:10.1109/aidas63860.2024.10729940 · Cited by 5
“Recent findings highlight the effectiveness of language models such as BERT, GPT, and XLNET, which often achieve state-of-the-art results [13].”
BackgroundRequirements Similarity and Retrieval
Abbas, M., Bashir, S., Saadatmand, M., Enoiu, E. P., & Sundmark, D. (2025) · Springer eBooks · doi:10.1007/978-3-031-73143-3_3 · Cited by 4
Building and Testing Fine-Grained Dataset of COVID-19 Tweets for Worry Prediction
Alharbi, T. S., & Fkih, F. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0130874 · Cited by 4
“, “I’m” instead of “I am”) were replaced by the original phrase as described in [37].”
BackgroundInfluentialRegional Knowledge: Sources, Representation and Management
Bordogna, G. (2022) · Springer eBooks · doi:10.1007/978-3-031-15648-9_2 · Cited by 4
Managing Healthcare Infodemic by deep learning in providing healthcare services
Ahmad, F. B., Umer, M., Sadiq, S., Majeed, R., Narducci, F., & Bisogni, C. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3610290 · Cited by 4
“Text analysis includes feature extraction methods [36], word representation [37], investigating syntactic and semantic aspects, and visualising the text [50].”
MethodTen propositions on machine learning in official statistics
van Delden, A., Burger, J., & Puts, M. (2023) · AStA Wirtschafts- und Sozialstatistisches Archiv · doi:10.1007/s11943-023-00330-0 · Cited by 5
“For text mining, Naseem et al. (2021) gave an overview of a number of ML algorithms and their pros and cons for use and under which conditions they are effective.”
Discourse Markers in Interaction: From Production to Comprehension
Li, R., Allein, L., Sileo, D., & Moens, M. (2022) · De Gruyter eBooks · doi:10.1515/9783110790351 · Cited by 4
Pretrained Bidirectional Distillation for Machine Translation
Zhuang, Y., & Tu, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.63 · Cited by 4
Alkhalil, B. F., Zhuang, Y., Mursi, K. T., & Aseeri, A. O. (2024) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi60790.2024.10585824 · Cited by 3
“However, it does show a slight performance gap when compared to domain-specific, fully-supervised SOTA models, where ”SOTA” refers to the latest leading models in sentiment analysis [14].”
BackgroundAnalysis Comparison of FastText and Word2vec for Detecting Offensive Language
Lumbantoruan, R., Siregar, R. U., Manik, I., Tambunan, N. D., & Simanjuntak, H. T. A. (2022) · International Conference of Computer Science and Information Technology (ICoSNIKOM) · doi:10.1109/icosnikom56551.2022.10034886 · Cited by 7
“This feature presentation technique provides a powerful tool for uncovering relationships in the corpus and determining token similarity [14].”
MethodFeng, S., Lubis, N., Ruppik, B., Geishauser, C., Heck, M. C., Lin, H.-C., van Niekerk, C., Vukovic, R., & Gašić, M. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.8 · Cited by 11
Pınarbaşı, F. (2024) · Nevşehir Hacı Bektaş Veli Üniversitesi SBE Dergisi · doi:10.30783/nevsosbilen.1508802 · Cited by 3
Metadata
- Authors
- Naseem, U., Razzak, I., Khan, S. K., & Prasad, M.
- Affiliations
- The University of Sydney (AU)
- Deakin University (AU)
- RMIT University (AU)
- University of Technology Sydney (AU)
- Venue
- ACM Transactions on Asian and Low-Resource Language Information Processing
- Published
- 2021-06-30
- Volume
- 20
- Issue
- 5
- Pages
- 1-35
- ISSN
- 2375-4699 (ISSN-L)
- 2375-4702
- OpenAlex
- W3095760691
- Cited by
- 194
- References
- 168
- Citations by year
- 2026: 22 · 2025: 31 · 2024: 37 · 2023: 61 · 2022: 32 · 2021: 10
- FWCI
- 12.26
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Asian and Low-Resource Language Information Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- ResearchOnline at James Cook University (James Cook University) (submittedVersion)
- arXiv (Cornell University) (submittedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Deakin Research Online (Deakin University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Figshare (submittedVersion)
- OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word representation, word embeddings, text representation, evaluation metrics, low-resource languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.