CoChat
Sign Up
Cited by 70Sentiment Analysis and Opinion MiningTopic ModelingStock Market Forecasting Methods

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Importance of Self-Attention for Sentiment Analysis

Open access
Letarte, Gaël, Paradis, Frédérik, Giguère, Philippe, Laviolette, François·2018·Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
DOI 10.18653/v1/w18-5429
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite their superior performance, deep learning models often lack interpretability.In this paper, we explore the modeling of insightful relations between words, in order to understand and enhance predictions.To this effect, we propose the Self-Attention Network (SANet), a flexible and interpretable architecture for text classification.Experiments indicate that gains obtained by self-attention is task-dependent.For instance, experiments on sentiment analysis tasks showed an improvement of around 2% when using selfattention compared to a baseline without attention, while topic classification showed no gain.Interpretability brought forward by our architecture highlighted the importance of neighboring word interactions to extract sentiment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/w18-5429

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Our Self-Attention Network (SANet), derived from the Transformer architecture (Vaswani et al., 2017). The self-attention block is repeated N times.
Figure 1 Our Self-Attention Network (SANet), derived from the Transformer architecture (Vaswani et al., 2017). The self-attention block is repeated N times.CC-BY · From the PDF
Figure 2 Visualization of sequences length distributions. For each dataset, the total number of examples is presented on the right and task semantics are identified on the left: Topic Classification (TC) or Sentiment Analysis (SA).
Figure 2 Visualization of sequences length distributions. For each dataset, the total number of examples is presented on the right and task semantics are identified on the left: Topic Classification (TC) or Sentiment Analysis (SA).CC-BY · From the PDF
Figure 3 Self-attention different behavior for each text classification task. The attention matrices were extracted from the SANet base model applied on the testing set of each dataset. Words on the y-axis are attending to the words on the x-axis. GT refers to the ground truth.
Figure 3 Self-attention different behavior for each text classification task. The attention matrices were extracted from the SANet base model applied on the testing set of each dataset. Words on the y-axis are attending to the words on the x-axis. GT refers to the ground truth.CC-BY · From the PDF
Figure 4 Randomly selected attention matrices for topic classification task. Each row corresponds to a different dataset in this order: AG's News , DBPedia and Yahoo! Answers . The column-based pattern is clearly present in the attention mechanism for topic classification.
Figure 4 Randomly selected attention matrices for topic classification task. Each row corresponds to a different dataset in this order: AG's News , DBPedia and Yahoo! Answers . The column-based pattern is clearly present in the attention mechanism for topic classification.CC-BY · From the PDF
Figure 5 Randomly selected attention matrices for sentiment analysis task. Each row corresponds to a different dataset in this order: Yelp Review Polarity , Yelp Review Full , Amazon Review Full and Amazon Review Polarity . The diagonal band pattern of the matrices is clearly present in the attention mechanism for sentiment analysis except for the Yelp Review Polarity dataset.
Figure 5 Randomly selected attention matrices for sentiment analysis task. Each row corresponds to a different dataset in this order: Yelp Review Polarity , Yelp Review Full , Amazon Review Full and Amazon Review Polarity . The diagonal band pattern of the matrices is clearly present in the attention mechanism for sentiment analysis except for the Yelp Review Polarity dataset.CC-BY · From the PDF
Figure 6 Positional encoding impact for sentiment gradation through self-attention mechanism. Both examples are extracted from the testing set of the Yelp Review Full dataset. Label 4 means 5-star review and label 0 means 1-star review. Words on the y-axis are attending to the words on the x-axis. GT refers to the ground truth.
Figure 6 Positional encoding impact for sentiment gradation through self-attention mechanism. Both examples are extracted from the testing set of the Yelp Review Full dataset. Label 4 means 5-star review and label 0 means 1-star review. Words on the y-axis are attending to the words on the x-axis. GT refers to the ground truth.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 25

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. · arXiv
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. · arXiv
  3. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyz- ing text with the natural language toolkit. " O'Reilly Media, Inc.".
  4. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733. · arXiv
  5. Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Be- rant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), volume 1, pages 209-220.
  6. Ruth C Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturba- tion. arXiv preprint arXiv:1704.03296. · arXiv
  7. Jeremy Howard and Sebastian Ruder. 2018. Fine- tuned language models for text classification. arXiv preprint arXiv:1801.06146. · arXiv
  8. Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, volume 1, page 3.
  9. Rie Johnson and Tong Zhang. 2016. Convolutional neural networks for text categorization: Shallow word-level vs. deep character-level. arXiv preprint arXiv:1609.00718. · arXiv
  10. Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categoriza- tion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), volume 1, pages 562-570.
  11. Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Net- work in network. arXiv preprint arXiv:1312.4400. · arXiv
  12. Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. 2015. Is object localization for free?-weakly- supervised learning with convolutional neural net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 685-694.
  13. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Empiri- cal Methods in Natural Language Processing, pages 2249-2255.
  14. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 confer- ence on empirical methods in natural language pro- cessing (EMNLP), pages 1532-1543.
  15. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144. ACM.
  16. Karen Simonyan, Andrea Vedaldi, and Andrew Zisser- man. 2013. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. · arXiv
  17. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, pages 6000-6010.
  19. Shiyao Wang, Minlie Huang, and Zhidong Deng. 2018. Densely connected cnn with multi-scale feature at- tention for text classification. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18.
  20. Sida Wang and Christopher D Manning. 2012. Base- lines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Lin- guistics: Short Papers-Volume 2, pages 90-94. As- sociation for Computational Linguistics.
  21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual at- tention. In International Conference on Machine Learning, pages 2048-2057.
  22. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchi- cal attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1480-1489.
  23. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Eu- ropean conference on computer vision, pages 818- 833. Springer.
  24. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in neural information pro- cessing systems, pages 649-657.
  25. Yizhe Zhang, Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, and Lawrence Carin. 2017. Decon- volutional paragraph representation learning. In Ad- vances in Neural Information Processing Systems, pages 4172-4182.

Cited by

Sorted by citations

Metadata

Authors
Letarte, G., Paradis, F., Giguère, P., & Laviolette, F.
Affiliations
Université Laval (CA)
ORCID
Venue
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
Publisher
Association for Computational Linguistics
Published
2018
Pages
267-275
DOI
10.18653/v1/w18-5429
OpenAlex
W2914713622
Cited by
70
References
5
Citations by year
2026: 3 · 2025: 2 · 2024: 9 · 2023: 13 · 2022: 7 · 2021: 18 · 2020: 10 · 2019: 8
FWCI
3.67
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) (publishedVersion, cc-by)
Funders
  • Nvidia
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
Keywords
self-attention, sentiment analysis, self-attention network, explainable deep learning, text classification, topic classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review