The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism
Open accessAbstract
There is a need to extract meaningful information from big data, classify it into different categories, and predict end-user behavior or emotions. Large amounts of data are generated from various sources such as social media and websites. Text classification is a representative research topic in the field of natural-language processing that categorizes unstructured text data into meaningful categorical classes. The long short-term memory (LSTM) model and the convolutional neural network for sentence classification produce accurate results and have been recently used in various natural-language processing (NLP) tasks. Convolutional neural network (CNN) models use convolutional layers and maximum pooling or max-overtime pooling layers to extract higher-level features, while LSTM models can capture long-term dependencies between word sequences hence are better used for text classification. However, even with the hybrid approach that leverages the powers of these two deep-learning models, the number of features to remember for classification remains huge, hence hindering the training process. In this study, we propose an attention-based Bi-LSTM+CNN hybrid model that capitalize on the advantages of LSTM and CNN with an additional attention mechanism. We trained the model using the Internet Movie Database (IMDB) movie review data to evaluate the performance of the proposed model, and the test results showed that the proposed hybrid attention Bi-LSTM+CNN model produces more accurate classification results, as well as higher recall and F1 scores, than individual multi-layer perceptron (MLP), CNN or LSTM models as well as the hybrid models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the other hand, long-term dependencies between word sequences can be captured by LSTM models, which is a vital requirement in the context of cyberbullying detection [125].”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [125]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“The model leverages the novelty of BERT (bidirectional encoder representation from transformers) [31] for text extraction.”
MethodConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · cited 2×
“To further increase the sentiment classification accuracy of unstructured text, Yoon Kim [7] proposed a 1D CNN approach for text classification.”
MethodFramewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 6,152
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · cited 1×
“With the 1D CNN, the convolution layer uses a 1D cross-correlation operation that involves a sliding convolution window with variable-size kernels over the input text starting from left to right [13].”
MethodRecurrent Convolutional Neural Networks for Text Classification
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9513 · Cited by 2,384
Deep learning for sentiment analysis: A survey
Zhang, L., Wang, S., & Liu, B. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1253 · Cited by 2,182 · cited 1×
“Natural-language-based sentiment classification has a wide range of applications, such as movie review classification, subjective and objective sentence classification, and text classification technology [1].”
BackgroundComparative Study of CNN and RNN for Natural Language ProcessingPreprint
Yin, W., Kann, K., Yu, M., & Schütze, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01923 · Cited by 1,264 · cited 1×
“Figure 2 shows the basic architecture of the 1D CNN [14].”
BackgroundBidirectional LSTM with attention mechanism and convolutional layer for text classification
Liu, G., & Guo, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.078 · Cited by 1,166 · cited 1×
“1D CNN processes text as a one-dimensional image and a 1D CNN is used to capture the latent associations between neighboring words, in contrast with LSTMs, which process each word in a sequential pattern [8].”
MethodA Convolutional Neural Network for Modelling SentencesPreprint
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.2188 · Cited by 967 · cited 1×
“With the 1D CNN, the convolution layer uses a 1D cross-correlation operation that involves a sliding convolution window with variable-size kernels over the input text starting from left to right [13].”
MethodTwitter Sentiment Analysis with Deep Convolutional Neural Networks
Severyn, A., & Moschitti, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767830 · Cited by 694 · cited 2×
“Early 2D CNNs have been widely used in image processing, but they have only recently been used in text classification tasks and have outperformed sequence-based approaches [12].”
Methodcontext2vec: Learning Generic Context Embedding with Bidirectional LSTM
Melamud, O., Goldberger, J., & Dagan, I. (2016) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1006 · Cited by 541
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max PoolingPreprint
Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., & Xu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06639 · Cited by 534 · cited 1×
“Unlike the LSTM network, the Bi-LSTM network has two parallel layers that propagate in two directions with forward and reverse passes to capture dependencies in two contexts [16,17].”
BackgroundText Classification Using Machine Learning Techniques
Ikonomakis, M., Kotsiantis, S. B., & Tampakas, V. T. (2005) · Cited by 490
Chinese comments sentiment classification based on word2vec and SVMperf
Zhang, D., Xu, H., Su, Z., & Xu, Y. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2014.09.011 · Cited by 399 · cited 1×
“Figure 1 describes the Word2vec process, which involves two distinct components: Continuous Bag of Words (CBOW) and skip-gram [10].”
BackgroundA Hybrid CNN-LSTM Model for Improving Accuracy of Movie Reviews Sentiment Analysis
Rehman, A. U., Malik, A. K., Raza, B., & Ali, W. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07788-7 · Cited by 353 · cited 1×
“Rehman et al. [24] proposed a hybrid model that uses a very deep CNN and LSTM to overcome previous challenges for sentiment analysis.”
BackgroundUnderstanding the use of Virtual Reality in Marketing: A text mining-based review
Loureiro, S. M. C., Guerreiro, J., Eloy, S., Langaro, D., & Panchapakesan, P. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.055 · Cited by 338 · cited 1×
“Recently, text classification has been applied in specialized fields such as sentiment analysis and clinical [33–35], law [36,37], healthcare [38–40], and business marketing [41,42].”
MethodA Novel Hybrid Deep Learning Model for Sentiment Classification
Salur, M. U., & Aydın, İ. (2020) · IEEE Access · doi:10.1109/access.2020.2982538 · Cited by 252 · cited 1×
“[28] proposed a novel hybrid model that combines different word embedding (Word2Vec, FastText, and character-level embedding) with various learning approaches (LSTM, Gated recurrent unit (GRU), Bi-LSTM, CNN).”
Pouyanfar, S., Yang, Y., Chen, S., Shyu, M., & Iyengar, S. S. (2018) · ACM Computing Surveys · doi:10.1145/3150226 · Cited by 195
Research on Text Classification Based on CNN and LSTM
Luan, Y., & Lin, S. (2019) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica.2019.8873454 · Cited by 191 · cited 1×
“The study in [25] proposed a text classification model called CNN-COIF-LSTM and experiments that included eight variants, showed that the combination of CNN and LSTM without an activation function or a variant thereof exhibited higher accuracy.”
MethodDensely Connected CNN with Multi-scale Feature Attention for Text Classification
Wang, S., Huang, M., & Deng, Z. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/621 · Cited by 181 · cited 3×
“…model has two major problems. in one fixed-size vector, there can be instances of Since all the information cannot be compressed in one fixed-size vector, there can be instances of information loss, and a vanishing gradient problem [18]. information loss, and a vanishing gradient problem [18].”
BackgroundMethodInfluentialZhang, J., Kowsari, K., Harrison, J. H., Lobo, J. M., & Barnes, L. E. (2018) · IEEE Access · doi:10.1109/access.2018.2875677 · Cited by 173 · cited 1×
“Recently, text classification has been applied in specialized fields such as sentiment analysis and clinical [33–35], law [36,37], healthcare [38–40], and business marketing [41,42].”
MethodOpinion mining using ensemble text hidden Markov models for text classification
Kang, M., Ahn, J., & Lee, K. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.07.019 · Cited by 150 · cited 1×
“Recently, text classification has been applied in specialized fields such as sentiment analysis and clinical [33–35], law [36,37], healthcare [38–40], and business marketing [41,42].”
MethodLSTM-CNN Hybrid Model for Text Classification
Zhang, J., Li, Y., Tian, J., & Li, T. (2018) · IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) · doi:10.1109/iaeac.2018.8577620 · Cited by 143 · cited 1×
“, it has demonstrated excellent performance for Therefore, it has demonstrated excellent”
BackgroundTree-Structured Regional CNN-LSTM Model for Dimensional Sentiment Analysis
Wang, J., Yu, L.-C., Lai, K.-H. R., & Zhang, X. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2959251 · Cited by 142 · cited 1×
“[26] proposed a local CNN-LSTM model with a tree structure to measure the steps in emotional analysis.”
MethodEvent Detection and Co-reference with Minimal Supervision
Peng, H., Song, Y., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1038 · Cited by 130 · cited 1×
“The CBOW component infers the target word when given the context words, while the skip-gram component infers the context words when given an input word [11].”
BackgroundText Classification Research with Attention-based Recurrent Neural Networks
Du, C., & Huang, L. (2018) · International Journal of Computers Communications & Control · doi:10.15837/ijccc.2018.1.3142 · Cited by 107 · cited 1×
“As an alternative, an attention mechanism has been proposed as an emerging technique for increasing the prediction accuracy, although this decreases when the input sequence becomes long [19].”
MethodText Classification Based on Hybrid CNN-LSTM Hybrid Model
She, X., & Zhang, D. (2018) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid.2018.10144 · Cited by 100 · cited 1×
“[27] have proposed a novel hybrid approach that leverages the capabilities of CNN to extract local features and addresses its inherent weakness to express long-term contextual features.”
BackgroundA High Efficient Biological Language Model for Predicting Protein–Protein Interactions
Wang, Y., You, Z., Yang, S., Xiao, L., Jiang, T.-H., & Zhou, X. (2019) · Cells · doi:10.3390/cells8020122 · Cited by 98
A Text Sentiment Classification Modeling Method Based on Coordinated CNN‐LSTM‐Attention Model
Zhang, Y., Zheng, J., Jiang, Y., Huang, G., & Chen, R. (2019) · Chinese Journal of Electronics · doi:10.1049/cje.2018.11.004 · Cited by 92
Research on Patent Text Classification Based on Word2Vec and LSTM
Xiao, L., Wang, G., & Zuo, Y. (2018) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid.2018.00023 · Cited by 68 · cited 2×
“[23] proposed a text classification model based on Word2vec and LSTM to efficiently classify text in the security field.”
MethodHierarchical Convolutional Attention Networks for Text Classification
Gao, S. B., Ramanathan, A., & Tourassi, G. D. (2018) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w18-3002 · Cited by 60 · cited 1×
“…equally; the part the input word that is However, not all the input sentences are referenced equally; the part of the input word that is related to the word to be predicted at that time receives more attention [20]. related to the word to be predicted at that time receives more attention [20].”
BackgroundA Fusion Model-Based Label Embedding and Self-Interaction Attention for Text Classification
Dong, Y., Liu, P., Zhu, Z., Wang, Q., & Zhang, Q. (2019) · IEEE Access · doi:10.1109/access.2019.2954985 · Cited by 53 · cited 1×
“[30] proposed a method that use a self-interaction attention approach and combined it with label embedding.”
MethodSentiment Analysis of Citations Using Word2vecPreprint
Liu, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00177 · Cited by 42 · cited 1×
“Word2vec is a popular sequence embedding method that transforms natural-language into distributed vector representations [9].”
MethodAC-BLSTM: Asymmetric Convolutional Bidirectional LSTM Networks for Text ClassificationPreprint
Liang, D., & Zhang, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01884 · Cited by 29 · cited 1×
“Bi-LSTM can learn long-term dependencies without retaining duplicate context information [15].”
BackgroundSeguí, F. L., Aguilar, R. A.-E., de Maeztu, G., García‐Altés, A., Cuyàs, F. G., Walsh, S. M., Castro, M. S., & Vidal‐Alaball, J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17031093 · Cited by 28
Law text classification using semi-supervised convolutional neural networks
Li, P., Zhao, F. Q., Li, Y., & Zhu, Z. (2018) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc.2018.8407150 · Cited by 23
Srivastava, S. K., Singh, S. K., & Suri, J. S. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-819445-4.00016-3 · Cited by 18
Jasmir, J., Nurmaini, S., Malik, R. F., & Abidin, D. Z. (2020) · Sriwijaya International Conference on Information Technology and Its Applications (SICONIAN) · doi:10.2991/aisr.k.200424.061 · Cited by 10 · cited 1×
“Although the CNN shrinks the input features to be used for prediction, the correlation between each word and final classification is not the same for all input words.”
MethodExemplar Auditing for Multi-Label Biomedical Text ClassificationPreprint
Schmaltz, A., & Beam, A. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03093 · Cited by 7
Research works printed but not linked: 2
- Pouyanfar, 2018, A survey on deep learning: Algorithms, techniques, and applications, ACM Comput. Surv. (CSUR), 51, 1
- Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional LSTM-CRF models for sequence tagging. arXiv.
Cited by
Sorted by citations
A Survey of Sentiment Analysis: Approaches, Datasets, and Future Research
Tan, K. L., Lee, C. P., & Lim, K. M. (2023) · Applied Sciences · doi:10.3390/app13074550 · Cited by 307
“Jang et al. (2020) [34] improved on the hybrid CNN and BiLSTM model by incorporating an attention mechanism.”
Shaik, T., Tao, X., Li, Y., Dann, C. E., McDonald, J., Redmond, P., & Galligan, L. (2022) · IEEE Access · doi:10.1109/access.2022.3177752 · Cited by 362
“…[172]–[174] Text Evaluation Text Summarization LSTM-NN, CNN, Bi-GRU, TextRank, LSA, BERT, LSTM-CNN [57], [92]–[109] LSTM, CNN, BLSTM-C [19], [110]–[114] Entity Extraction CNN [115]–[117] Knowledge Graphs Multidimensional knowledge graphs [118], [119] CNN, LSTM [3], [120]–[129] implementation…”
MethodImpact of word embedding models on text analytics in deep learning environment: a review
Asudani, D. S., Nagwani, N. K., & Singh, P. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10419-1 · Cited by 274
“The construction of text classification and document classification techniques includes extracting features, dimension minimization, classifier selection, and assessments (Jang et al. 2020).”
MethodWearable Sensor-Based Human Activity Recognition with Transformer Model
Dirgová Luptáková, I., Kubovčík, M., & Pospíchal, J. (2022) · Sensors · doi:10.3390/s22051911 · Cited by 270
“Before the transformer model was introduced, the cutting-edge systems for complex time series problems such as natural language processing were based on gated recurrent neural networks with a superimposed attention mechanism [42].”
MethodBaviskar, D., Ahirrao, S. N., Potdar, V. M., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3072900 · Cited by 175
“We found few other studies also using LSTM and GRU for information extraction task [29], [50], [57], [68], [110].”
MethodResult
Metadata
- Authors
- Jang, B., Kim, M., Harerimana, G., Kang, S.-U., & Kim, J. W.
- Affiliations
- Sangmyung University (KR)
- Corresponding author
- Kim, Jong Wook
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2020-08-24
- Volume
- 10
- Issue
- 17
- Pages
- 5841
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3080359515
- Cited by
- 442
- References
- 33
- Citations by year
- 2026: 25 · 2025: 93 · 2024: 97 · 2023: 101 · 2022: 73 · 2021: 50 · 2020: 3
- FWCI
- 22.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Research Foundation of Korea (NRF-2019R1F1A1058058)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- text classification, bidirectional LSTM, convolutional neural networks, attention mechanism, Word2Vec, IMDB movie reviews, sentiment analysis, feature extraction, long short-term memory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
