Learning to Generate Reviews and Discovering Sentiment
Open accessAbstract
We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multi-task objectives may be more effective than single-task ones because they impose more task constraints on the representation, leading to a smaller and higher-quality solution space.
“Such multi-task objectives may be more effective than single-task ones (e.g., ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space (Chen et al., 2020;He et al., 2020;Radford et al., 2017;2019).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Radford et al., 2017
The Introduction (pp. 1–2) discusses representation-learning objectives, and the Discussion (p. 7) says an unmodified language-modeling objective can learn high-quality representations. Neither passage addresses whether multi-task objectives outperform single-task objectives by constraining representations or narrowing the solution space. Searches for those terms and their paraphrases found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1× · see citing works
“Image classifiers learn a broadly useful hierarchy of feature detectors re-representing raw pixels as edges, textures, and objects (Zeiler & Fergus, 2014).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · cited 1× · see citing works
“On the MR (Pang & Lee, 2005) and CR (Hu & Liu, 2004) sentiment analysis datasets we improve the state of the art by a significant margin.”
MethodExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1× · see citing works
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · cited 1× · see citing works
“Pre-trained word vectors are a vital part of many modern NLP systems (Collobert et al., 2011).”
BackgroundRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 2× · see citing works
“The Stanford Sentiment Treebank (SST) (Socher et al., 2013) was created specifically to evaluate more complex compositional models of language.”
MethodGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642 · cited 1× · see citing works
“There is also a long history of unsupervised representation learning (Olshausen & Field, 1997).”
BackgroundLearning and Transferring Mid-level Image Representations Using Convolutional Neural Networks
Oquab, M., Bottou, L., Laptev, I., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.222 · Cited by 3,150 · cited 1× · see citing works
“In the field of computer vision, it is now commonplace to reuse these representations on a broad suite of related tasks - one of the most successful examples of transfer learning to date (Oquab et al., 2014).”
BackgroundImproved Semantic Representations From Tree-Structured Long Short-Term Memory Networks
Tai, K. S., Socher, R., & Manning, C. D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1150 · Cited by 3,081 · see citing works
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · Cited by 2,913 · see citing works
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · Cited by 2,760 · see citing works
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural NetworksPreprint
Salimans, T., & Kingma, D. P. (2016) · Neural Information Processing Systems · doi:10.48550/arxiv.1602.07868 · Cited by 2,284 · see citing works
A Fast and Accurate Dependency Parser using Neural Networks
Chen, D., & Manning, C. D. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1082 · Cited by 1,810 · cited 1× · see citing works
“These representations, learned by modeling word co-occurrences, increase the data efficiency and generalization capability of NLP systems (Pennington et al., 2014) (Chen & Manning, 2014).”
BackgroundAnnotating Expressions of Opinions and Emotions in Language
Wiebe, J., Wilson, T., & Cardie, C. (2005) · Computers and the Humanities · doi:10.1007/s10579-005-7880-9 · Cited by 1,763 · see citing works
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 1× · see citing works
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundExploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · cited 2× · see citing works
“As discussed in Jozefowicz et al. (2016) performance on these datasets is primarily dominated by regularization.”
BackgroundUnsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F., Boureau, Y.-L., & LeCun, Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383157 · Cited by 1,186 · see citing works
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., & Socher, R. (2015) · arXiv · doi:10.48550/arxiv.1506.07285 · Cited by 1,020 · see citing works
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 1× · see citing works
“In contrast to learning a generic representation on one large dataset and then evaluating on other tasks/datasets, Dai & Le (2015) proposed using similar unsupervised objectives such as sequence autoencoding and language modeling to first pretrain a model on a dataset and then finetune it for a given…”
MethodVisualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · cited 1× · see citing works
“Previous work analysing LSTM language models showed the existence of interpretable units that indicate position within a line or presence inside a quotation (Karpathy et al., 2015).”
BackgroundObject Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · cited 1× · see citing works
“Analysis of the task specific representations learned by these models reveals many fascinating properties (Zhou et al., 2014).”
BackgroundToward Controlled Generation of TextPreprint
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00955 · Cited by 802 · cited 1× · see citing works
“Hu et al. (2017) and Dong et al. (2017) both designed conditional generative models to disentangle the content of text from various attributes like sentiment or tense.”
BackgroundPang, B., & Lee, L. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0506075 · Cited by 744 · cited 1× · see citing works
“On the MR (Pang & Lee, 2005) and CR (Hu & Liu, 2004) sentiment analysis datasets we improve the state of the art by a significant margin.”
MethodUnsupervised construction of large paraphrase corpora: exploiting massively parallel news sources
Dolan, B., Quirk, C., & Brockett, C. (2004) · Proceedings of the 20th international conference on Computational Linguistics - COLING '04 · doi:10.3115/1220355.1220406 · Cited by 728 · cited 2× · see citing works
“The package received was blank and has no barcode.”
BackgroundMethodAdversarial Training Methods for Semi-Supervised Text ClassificationPreprint
Miyato, T., Dai, A. M., & Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07725 · Cited by 713 · cited 2× · see citing works
“One exception is that we use an L1 penalty for text classification results instead of L2 as we found this performed better in the very low data regime.”
BackgroundMethodInfluentialLearning Distributed Representations of Sentences from Unlabelled Data
Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566 · cited 2× · see citing works
“Hill et al. (2016) also raises concern about current evaluation tasks in their recent work which provides a thorough survey of architectures and objectives for learning unsupervised sentence representations - including the above mentioned skip-thoughts.”
BackgroundUnderstanding Neural Networks through Representation ErasurePreprint
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 493 · cited 2× · see citing works
“Models such as word2vec have also been observed to have small subsets of dimensions strongly associated with specific tasks (Li et al., 2016).”
BackgroundMarelli, M., Bentivogli, L., Baroni, M., Raffaella Bernardi, B., Menini, S., & Zamparelli, R. (2014) · International Workshop on Semantic Evaluation (SemEval) · doi:10.3115/v1/s14-2001 · Cited by 378 · cited 1× · see citing works
“While our model performs competitively on Microsoft Research Paraphrase Corpus (Dolan et al., 2004) in Table 3, it performs poorly on the SICK semantic relatedness task (Marelli et al., 2014) in Table 4.”
MethodTowards Universal Paraphrastic Sentence EmbeddingsPreprint
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08198 · Cited by 324 · see citing works
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · cited 6× · see citing works
“By inspecting the relative contributions of features on various datasets, we discovered a single unit within the mLSTM that directly corresponds to sentiment .”
MethodInfluentialUTF-8, a transformation format of ISO 10646
Yergeau, F. (1998) · doi:10.17487/rfc2279 · Cited by 220 · cited 1× · see citing works
“Our model processes text as a sequence of UTF-8 encoded bytes (Yergeau, 2003).”
MethodLearning to Generate Product Reviews from Attributes
Dong, L., Huang, S., Wei, F., Lapata, M., Zhou, M., & Xu, K. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1059 · Cited by 190 · cited 1× · see citing works
“Hu et al. (2017) and Dong et al. (2017) both designed conditional generative models to disentangle the content of text from various attributes like sentiment or tense.”
BackgroundTopicRNN: A Recurrent Neural Network with Long-Range Semantic DependencyPreprint
Dieng, A. B., Wang, C., Gao, J., & Paisley, J. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01702 · Cited by 182 · cited 2× · see citing works
“Combining language modelling with topic modelling and fitting a small supervised feature extractor on top has also achieved strong results on in-domain document level sentiment analysis (Dieng et al., 2016).”
MethodEnsemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie ReviewsPreprint
Mesnil, G., Mikolov, T., Ranzato, M., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5335 · Cited by 102 · cited 1× · see citing works
“Fitting a threshold to this single unit achieves a test accuracy of 92.30% which outperforms a strong supervised results on the dataset, the 91.87% of NB-SVM trigram (Mesnil et al., 2014), but is still below the semi-supervised state of the art of 94.09% (Miyato et al., 2016).”
BackgroundInfluentialDeep Learning with Dynamic Computation GraphsPreprint
Looks, M., Herreshoff, M., Hutchins, D. S., & Norvig, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02181 · Cited by 88 · cited 1× · see citing works
“The representation learned by our model achieves 91.8% significantly outperforming the state of the art of 90.2% by a 30 model ensemble (Looks et al., 2017).”
MethodInfluentialInferring Networks of Substitutable and Complementary ProductsPreprint
McAuley, J., Pandey, R., & Leskovec, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08839 · Cited by 87 · cited 2× · see citing works
“The model chosen for the large scale experiment is a single layer multiplicative LSTM (Krause et al., 2016) with 4096 units.”
Method
Flagged first, then by citations
3 of the 47 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.
- Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin. Reexamining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 182-190. Association for Computational Linguistics, 2012.
- Munkhdalai, Tsendsuren and Yu, Hong. Neural semantic encoders. arXiv preprint arXiv:1607.04315 , 2016. · arXiv
- Ng, Andrew Y. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twentyfirst international conference on Machine learning , pp. 78. ACM, 2004.
- Pang, Bo and Lee, Lillian. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics , pp. 271. Association for Computational Linguistics, 2004.
- Wang, Sida and Manning, Christopher D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2 , pp. 90-94. Association for Computational Linguistics, 2012.
- Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems , pp. 649-657, 2015.
- Zhao, Han, Lu, Zhengdong, and Poupart, Pascal. Selfadaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070 , 2015. · arXiv
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
“Skip-thought vectors [26] and Byte mLSTM [27] are alternatives which go beyond word prediction with a Recurrent Neural Network, and use maximum likelihood over sequences of observations.”
BackgroundMethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
“…modeling can be seen as the ideal source task and a counter-part of ImageNet for NLP: It captures many facets of language relevant for downstream tasks, such as long-term dependencies (Linzen et al., 2016), hierarchical relations (Gulordava et al., 2018), and sentiment (Radford et al., 2017).”
BackgroundPei, K., Cao, Y., Yang, J., & Jana, S. (2017) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3132747.3132785 · Cited by 1,296 · see citing works
“It has recently been shown that each neuron in a DNN tends to independently extract a specific feature of the input instead of collaborating with other neurons for feature extraction [58, 87].”
BackgroundUnified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · see citing works
Contextual String Embeddings for Sequence Labeling
Akbik, A., Blythe, D. A. J., & Vollgraf, R. (2018) · International Conference on Computational Linguistics · Cited by 1,084 · see citing works
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · see citing works
“This success has been fueled by pretraining on large quantities of raw text using pretext objectives, such as language modeling (Radford et al., 2017) or word vectors (Devlin et al., 2019), that require no supervision.”
BackgroundThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
SemEval-2018 Task 1: Affect in Tweets
Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1001 · Cited by 746 · see citing works
“• Sentiment neurons (Radford et al., 2017): a byte-level recurrent language model for learning sentence representations.”
BackgroundFelbo, B., Mislove, A. E., Søgaard, A., Rahwan, I., & Lehmann, S. (2017) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 712 · see citing works
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · see citing works
“…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
InfluentialLow-N protein engineering with data-efficient deep learning
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., & Church, G. M. (2021) · Nature Methods · doi:10.1038/s41592-021-01100-y · Cited by 422 · see citing works
Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation ModelsPreprint
Guimaraes, G. L., Sánchez-Lengeling, B., Outeiral, C., Farias, P. L. C., & Aspuru‐Guzik, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10843 · Cited by 586 · see citing works
“In natural language processing, a given sentiment might be emphasized, maybe for producing movie reviews [Radford et al., 2017].”
BackgroundFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · see citing works
“Since the reward model needs to understand language, we initialize it as a random linear function of the final embedding output of the language model policy ρ following Radford et al. (2018) (see section 4.”
BackgroundMethodSelf-citationUnderstanding the role of individual units in a deep neural network
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., & Torralba, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907375117 · Cited by 375 · see citing works
“Intriguingly, within state-of-the-art deep networks, it has been observed that many single units match human-interpretable concepts that were not explicitly taught to the network: units have been found to detect objects, parts, textures, tense, gender, context, and sentiment (1–7).”
BackgroundAnalysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403 · see citing works
“PDA [84] image feature importance local, ante-hoc DVE [83] image visualization local, post-hoc Grad-CAM [76] image saliency mask local, post-hoc Grad-CAM++ [78] image saliency mask local, post-hoc Smooth-Grad [86] image sensitivity analysis local, ante-hoc ME [88] image visualization local, post-hoc Summit [89] image…”
MethodA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. F. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 344 · see citing works
“, 2017), GANs (Wang & Wan, 2018) or Seq2Seq models (Radford et al., 2017).”
Fine-tuned Language Models for Text Classification.Preprint
Howard, J., & Ruder, S. (2018) · arXiv (Cornell University) · Cited by 357 · see citing works
Deep Semantic Feature Learning for Software Defect Prediction
Wang, S., Liu, T., Nam, J., & Tan, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877612 · Cited by 319 · see citing works
“[79] focused on understanding the properties of representations learned by byte-level recurrent language models for sentiment analysis.”
BackgroundWieting, J., & Gimpel, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1042 · Cited by 318 · see citing works
Exposing the Fingerprint: Dissecting the Impact of the Wireless Channel on Radio Fingerprinting
Al-Shawabka, A., Restuccia, F., D’Oro, S., Jian, T., Rendon, B. C., Soltani, N., Dy, J., Ioannidis, S., Chowdhury, K. R., & Melodia, T. (2020) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom41043.2020.9155259 · Cited by 312 · see citing works
“…in more mature and widely-explored learning domains such as computer vision and natural language processing, where massive-scale labeled datasets such as MNIST for image classification [20] and IMDb for sentiment analysis [21] have been completely available to the research community for many years.”
BackgroundChallenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288 · see citing works
“research on modeling empathetic interactions in conversation [100]. We will present dialog datasets in Section7. Controlling the emotion or sentiment has become a popular topic in language generation [34, 45, 97]. In [97], an RNN-based language model is trained on large-scale review data where some neurons are reporte…”
MethodZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020) · Distill · doi:10.23915/distill.00024.001 · Cited by 288 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286 · see citing works
Controlling Linguistic Style Aspects in Neural Language Generation
Ficler, J., & Goldberg, Y. (2017) · arXiv.org · doi:10.18653/v1/w17-4912 · Cited by 432 · see citing works
“…the content of the generated text (Lipton et al., 2015; Kid-don et al., 2016; Lebret et al., 2016; Kiddon et al., 2016; Tang et al., 2016; Radford et al., 2017), while a few model more stylistic aspects of the generated text such as the identity of the speaker in a dialog setting (Li et…”
BackgroundUnsupervised Pretraining for Sequence to Sequence Learning
Ramachandran, P., Liu, P., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1039 · Cited by 251 · see citing works
“Dai and Le (2015); Radford et al. (2017) are amongst the rare studies which showed the benefits of pretraining in a semi-supervised learning setting.”
BackgroundA Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 264 · see citing works
“…interesting tasks related to NLP like: text games [222], language to program execution [223], extraction of information [224], sentiment analysis [225], knowledge graph reasoning [226], automatic query reformulation [227], information retrival [228], summarization [229], [230] and semantic…”
BackgroundThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248 · see citing works
“LLM representations have also been observed to encode emotion and sentiment [36, 64].”
BackgroundSentiment Analysis for Fake News Detection
Alonso, M. Á., Vilares, D., Gómez‐Rodríguez, C., & Vilares, J. (2021) · Electronics · doi:10.3390/electronics10111348 · Cited by 253 · see citing works
“For instance, Radford et al. [141] showed how pre-training an LSTM on a large data set of Amazon reviews at the character level (byte level) could obtain state-of-the-art results for SA on standard data sets by simply fine-tuning on a few supervised examples.”
BackgroundPretraining-Based Natural Language Generation for Text Summarization
Zhang, H., Cai, J., Xu, J., & Wang, J. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1074 · Cited by 247 · see citing works
Deep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230 · see citing works
“1) Sentiment Analysis [93]–[95]: The input is a sequence of words, and the output is a single sentiment (positive, negative, or neutral).”
BackgroundGeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220 · see citing works
“Natural language generation has seen great progress with the advent of Transformers (Vaswani et al., 2017) and large scale training (Radford et al., 2017, 2018, 2019; Brown et al., 2020).”
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222 · see citing works
“tive LSTM (mLSTM) network on 80 million Amazon product reviews to extract 4096-dimension character-level features (Radford et al., 2017)3.”
BackgroundMethodA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
“Finally, it is worth mentioning that [81] presents the discovery that a single neuron unit of a DNN can perform alone a sentiment analysis task after the training of the network reaching the same level of performance of strong baselines.”
BackgroundEmotion Recognition on Twitter: Comparative Study and Training a Unison Model
Colnerič, N., & Demšar, J. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2807817 · Cited by 183 · see citing works
“Radford et al. [14] even reported on discovering the sentiment unit while training to generate reviews.”
BackgroundFrom attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181 · see citing works
ArtEmis: Affective Language for Visual Art
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01140 · Cited by 161 · see citing works
“Radford and colleagues [49] discovered that a single unit in recurrent language models trained without sentiment labels, is automatically learning concepts of sentiment; enabling sentiment-oriented manipulation by fixing the sign of that unit.”
BackgroundCompiler fuzzing through deep learning
Cummins, C., Petoumenos, P., Murray, A. C., & Leather, H. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213848 · Cited by 157 · see citing works
“Our approach is inspired by breakthrough results in modeling challenging and high dimensional datasets through unsupervised learning [4, 27, 28].”
MethodLearning to Compose Task-Specific Tree Structures
Choi, J., Yoo, K. M., & Lee, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11975 · Cited by 8 · see citing works
“…NSE (Munkhdalai and Yu, 2017a) 89.7 52.8 CT-LSTM (single) (Looks et al., 2017) 89.4 52.3 CT-LSTM (ensemble) (Looks et al., 2017) 90.2 53.6 byte-mLSTM (Radford et al., 2017) 91.8 52.9 BCN+Char+CoVe (McCann et al., 2017) 90.3 53.7 The result shows that our model effectively maps similar sentences…”
InfluentialInductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT
Li, X., & Fourches, D. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-00430-x · Cited by 145 · see citing works
Canine : Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00448 · Cited by 161 · see citing works
On Completeness-aware Concept-Based Explanations in Deep Neural NetworksPreprint
Yeh, C., Kim, B., Arık, S. Ö., Li, C., Pfister, T., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07969 · Cited by 144 · see citing works
“Some use dimensionality reduction methods [34, 35], while others uncover higher level human-relatable concepts by dimensionality reduction (e.g. for speech [36] and language [37, 38]).”
Sekaran, K., Chandana, P., Krishna, N. M., & Kadry, S. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7419-5 · Cited by 133 · see citing works
“The Results of the prediction of cancer with respect to the threshold parameters fed to the Deep learning is presented by identifying the tumor [14] rate of spread in the head section of the pancreas, after diagnosing and treatment, the spread of tumor size has been decreased, which helped patient to get cured than th…”
BackgroundWhat Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., & Glass, J. R. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33016309 · Cited by 127 · see citing works
“Some recent work on neural language models and machine translation analyzes specific neurons of length (Qian, Qiu, and Huang 2016a; Shi, Knight, and Yuret 2016) and sentiment (Radford, Jozefowicz, and Sutskever 2017).”
BackgroundExplainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
Lin, Y., Ren, P., Chen, Z., Ren, Z., Ma, J., & de Rijke, M. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2906190 · Cited by 126 · see citing works
An Analysis of Neural Language Modeling at Multiple ScalesPreprint
Merity, S., Keskar, N. S., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08240 · Cited by 125 · see citing works
“…LMs are useful in many applications including speech recognition (Yu & Deng, 2014), machine translation (Koehn, 2009), natural language generation (Radford et al., 2017; Merity et al., 2017a), learning token embeddings, and as a general-purpose feature extractor for downstream tasks (P. Matthew,…”
BackgroundLAMOL: LAnguage MOdeling for Lifelong Language LearningPreprint
Sun, F.-K., Ho, C.-H., & Lee, H.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03329 · Cited by 124 · see citing works
Learning Language Representations for Typology Prediction
Malaviya, C., Neubig, G., & Littell, P. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1268 · Cited by 123 · see citing works
“sentiment (Radford et al., 2017), we expect that the cell states will represent features that may be”
BackgroundIdentifying and Controlling Important Neurons in Neural Machine TranslationPreprint
Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. R. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1811.01157 · Cited by 206 · see citing works
“In contrast, previous work found meaningful individual neurons in computer vision (Zeiler & Fergus, 2014; Zhou et al., 2016; Bau et al., 2017, among others) and in a few NLP tasks (Karpathy et al., 2015; Radford et al., 2017; Qian et al., 2016a).”
BackgroundResult(2019) · doi:10.18653/v1/n19-1002 · Cited by 118 · see citing works
“Radford et al. (2017), for example, detected a “sentiment” grandmother cell in a language-model-trained network.”
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · see citing works
SEDAT: Sentiment and Emotion Detection in Arabic Text Using CNN-LSTM Deep Learning
Abdullah, M., Hadzikadicy, M., & Shaikhz, S. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00134 · Cited by 111 · see citing works
“In this system, we uses word and document embedding models with feature vectors extracted from Arabic and translated tweets by using the AffectiveTweets package, Deepmoji, and Unsupervised Sentiment Neurons.”
MethodEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
“Radford et al. (2017, 2019) and Brown et al. (2020) demonstrated that language models trained with an unsupervised objective can perform a variety of tasks out-of-the-box.”
BackgroundAdelani, D. I., Mai, H., Fang, F., Nguyen, H. H., Yamagishi, J., & Echizen, I. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-44041-1_114 · Cited by 108 · see citing works
“further propose a “skill-up” method in which an LM is explicitly conditioned by a specified sentiment. This method requires a natural language processing expert to train a tailored LM. Radford et al. [31] reported that a sentiment neuron can be learned by using a single-layer multiplicative LSTM (mLSTM) [28]. The senti…”
MethodXia, Y., Xiao, Z., Jazdi, N., & Weyrich, M. (2024) · IEEE Access · doi:10.1109/access.2024.3415470 · Cited by 99 · see citing works
“The proposed ‘‘semantic node’’ is specified in Table 2 and visualized in Figure 10 .”
BackgroundGenerating and designing DNA with deep generative modelsPreprint
Killoran, N., Lee, L. J., Delong, A., Duvenaud, D., & Frey, B. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06148 · Cited by 131 · see citing works
“[35, 36] demonstrated that one can sometimes inspect the activations of RNNs trained in an unsupervised manner and find neuron units whose activation captures some property of interest in the sequence.”
BackgroundVariational Pretraining for Semi-supervised Text Classification
Gururangan, S., Dang, T., Card, D., & Smith, N. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1590 · Cited by 95 · see citing works
“, 2000), but larger and more reliable gains have been obtained by using contextual embeddings trained with a language modeling (LM) objective on massive amounts of text from domains such as Wikipedia or news (Peters et al., 2018a; Devlin et al., 2019; Radford et al., 2018; Howard and Ruder, 2018).”
BackgroundPrivFT: Private and Fast Text Classification With Homomorphic Encryption
Badawi, A. A., Hoang, L., Mun, C. F., Laine, K., & Aung, K. M. M. (2020) · IEEE Access · doi:10.1109/access.2020.3045465 · Cited by 96 · see citing works
“Much of the recent NLP research has focused on transfer learning techniques such as pre-training word embeddings [37], [38], or pre-training language models on larger datasets and fine-tuning them for task-specific learning [5]–[8].”
BackgroundMethodxLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299 · see citing works
Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline
Ethayarajh, K. (2018) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w18-3012 · Cited by 93 · see citing works
Language Models as Agent Models
Andreas, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.423 · Cited by 92 · see citing works
Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
Shi, W., Wang, X., Oh, Y. J., Zhang, J., Sahay, S., & Yu, Z. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376843 · Cited by 93 · see citing works
“Character information was calculated by a pretrained char embedding [38].”
MethodBaşarslan, M. S., & Kayaalp, F. (2023) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-022-00386-3 · Cited by 88 · see citing works
Extractive Opinion Summarization in Quantized Transformer Spaces
Angelidis, S., Amplayo, R. K., Suhara, Y., Wang, X., & Lapata, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00366 · Cited by 85 · see citing works
“We use a Centroid method that encodes the entity’s reviews with BERT (average token vector; Devlin et al., 2019) or SentiNeu-ron (Radford et al., 2017), and picks the one closest to the mean review vector.”
MethodAlonso, M. Á., Vilares, D., Gómez‐Rodríguez, C., & Vilares, J. (2013) · Electronics · doi:10.3390/electronics · Cited by 82 · see citing works
Berge, G. T., Granmo, O., Tveit, T. O., Goodwin, M., Jiao, L., & Matheussen, B. V. (2019) · IEEE Access · doi:10.1109/access.2019.2935416 · Cited by 81 · see citing works
“…adversarial training (S. Wang and Manning 2012; Mesnil et al. 2014; Zhao 2016; Johnson and Zhang 2015; Dieng et al. 2016; Miyato, Dai, and Goodfellow 2016; Sachan, Zaheer, and Salakhutdinov 2018; Radford, Jozefowicz, and Sutskever 2017; Gray, Radford, and Kingma 2017; Howard and Ruder 2018).”
BackgroundAugmenting Neural Networks with First-order Logic
Li, T., & Srikumar, V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1028 · Cited by 78 · see citing works
“Node semantics may be assigned during model design (e.g. attention), or incidentally discovered in post hoc analysis (e.g., Le et al., 2012; Radford et al., 2017, and others).”
Do Convolutional Networks need to be Deep for Text Classification ?Preprint
Le, H. T., Cerisara, C., & Denis, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04108 · Cited by 84 · see citing works
Controllable Text Generation.Preprint
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017) · arXiv (Cornell University) · Cited by 73 · see citing works
Anomaly Detection in Operating System Logs with Deep Learning-Based Sentiment Analysis
Studiawan, H., Sohel, F. A., & Payne, C. (2020) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2020.3037903 · Cited by 69 · see citing works
Protein design via deep learning
Ding, W., Nakai, K., & Gong, H. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac102 · Cited by 66 · see citing works
“Although the data mining ability of RNN architecture used by UniRep might be inferior compared with current popular ones in the field of nature language processing like transformer, the basic conceptions it came up with and the impressive extensions it showed still influenced following researches a lot.”
InfluentialDeep Learning for Cognitive Neuroscience
Storrs, K. R., & Kriegeskorte, N. (2020) · The MIT Press eBooks · doi:10.7551/mitpress/11442.003.0077 · Cited by 68 · see citing works
hULMonA: The Universal Language Model in Arabic
Jundi, O. E., Antoun, W., Droubi, N. E., Hajj, H. M., El-Hajj, W., & Shaban, K. B. (2019) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/w19-4608 · Cited by 71 · see citing works
“…to obtain general understanding of a particular language due to its ability of capturing many aspects of language relevant for downstream tasks, such as long-term dependencies (Linzen et al., 2016), hierarchical relations (Gulordava et al., 2018), and sentiment orientation (Radford et al., 2017).”
BackgroundUnsupervised Opinion Summarization with Noising and Denoising
Amplayo, R. K., & Lapata, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.175 · Cited by 66 · see citing works
“, 2017), a centroidbased method which represents the input as IDFweighted word embeddings and selects as summary the review closest to the centroid, and (c) SENTINEURON, which is similar to WORD2VEC but uses a language model called Sentiment Neuron (Radford et al., 2017) as input representation.”
MethodDeep Learning for Software Defect Prediction
Omri, S., & Sinz, C. (2020) · doi:10.1145/3387940.3391463 · Cited by 62 · see citing works
“Radford et al. [42] focus on understanding the properties of representations learned by byte-level recurrent language models for sentiment analysis.”
BackgroundProceedings of the First Workshop on Abusive Language Online
(2017) · doi:10.18653/v1/w17-30 · Cited by 63 · see citing works
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1797 · Cited by 117 · see citing works
“Therefore, we alternate between supervised training on the QA pairs and pretraining on Open Web Text (Radford et al., 2017).”
MethodDeep learning frameworks for protein–protein interaction prediction
Hu, X., Feng, C., Ling, T., & Chen, M. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.06.025 · Cited by 58 · see citing works
“The architectures of UniRep contain a 1,900-dimensional single-layer multiplicative long short-term-memory (LSTM) recurrent neural networks (RNNs) [59], a four-layer stacked multiplicative LSTM of 256 dimensions and a four-layer stacked multiplicative of 64 dimensions.”
MethodCharacter-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works
“Character-level modeling has shown promise in many areas such as sentiment analysis [35], question answering [21] and classification [50], and is an exciting area due to its simplicity and the ability to easily adapt to other languages.”
BackgroundFleming, R. W., & Storrs, K. R. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.07.004 · Cited by 53 · see citing works
Story Centaur: Large Language Model Few Shot Learning as a Creative Writing Tool
Swanson, B., Mathewson, K. W., Pietrzak, B., Chen, S., & Dinalescu, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.29 · Cited by 56 · see citing works
“While most work concerned itself primarily with topping each other’s GLUE and SUPERGLUE scores (Wang et al., 2019), work from OpenAI kept the torch lit for investigating the emergent abilities of representation learning (Radford et al., 2017, 2018, 2019; Brown et al., 2020).”
BackgroundDeep Learning-Based Sentiment Classification: A Comparative Survey
Mabrouk, A., Redondo, R. P. D., & Kayed, M. (2020) · IEEE Access · doi:10.1109/access.2020.2992013 · Cited by 61 · see citing works
“Word embedding handles many SA tasks such as aspect term extraction [64], SC [32], [59], [65], [66], among other.”
BackgroundMethodInfluentialSimilarity Analysis of Contextual Word Representation Models
Wu, J., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.422 · Cited by 52 · see citing works
“While neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons (Qian et al., 2016; Shi et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Bau et al., 2019; Dalvi et al., 2019; Baan et al., 2019).”
BackgroundPinheiro, L. D. S., & Dras, M. (2017) · Cited by 54 · see citing works
SeerNet at SemEval-2018 Task 1: Domain Adaptation for Affect in Tweets
Duppada, V., Jain, R., & Hiray, S. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1002 · Cited by 51 · see citing works
“(Radford et al., 2017) developed an unsupervised system which learned an excellent representation of sentiment.”
BackgroundUnderstanding Pre-trained BERT for Aspect-based Sentiment Analysis
Xu, H., Shu, L., Yu, P., & Liu, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.21 · Cited by 53 · see citing works
“This is in contrast with the findings of sentiment neuron in pre-trained casual LMs (Radford et al., 2017) and we suspect the small hidden space (768) does not allow such a single neuron to exist.”
ResultInfluentialTechnology Solutions to Combat Online Harassment
Kennedy, G., McCollough, A. W., Dixon, E., Bastidas, A., Ryan, J. B., Loo, C., & Sahay, S. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3011 · Cited by 51 · see citing works
“(Radford et al., 2017) and have the potential to resist the ”adversarial” tokens described in (Hosseini et al., 2017).”
BackgroundStojanovski, D., Strezoski, G., Madjarov, G., Dimitrovski, I., & Chorbev, I. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6168-1 · Cited by 50 · see citing works
“[38] showed that training a language model in an unsupervised manner can also be used for sentiment analysis.”
BackgroundChemlistem: chemical named entity recognition using recurrent neural networks
Corbett, P. T., & Boyle, J. (2018) · Journal of Cheminformatics · doi:10.1186/s13321-018-0313-8 · Cited by 48 · see citing works
“Radford et al. [16] trained a byte-level language model on product reviews, and were able to use this model to train a sentiment analysis system with high data efficiency.”
MethodYou, Q., Jin, H., & Luo, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10121 · Cited by 47 · see citing works
“Recently, a sentiment unit is discovered in a RNN based generation model (Radford, Jozefowicz, and Sutskever 2017).”
BackgroundEcco: An Open Source Library for the Explainability of Transformer Language Models
Alammar, J. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.30 · Cited by 46 · see citing works
“Previous work (Karpathy et al., 2015; Strobelt et al., 2017; Poerner et al., 2018; Radford et al., 2017; Olah et al., 2017, 2018; Bau et al., 2018; Dalvi et al., 2019; Rethmeier et al., 2020) has examined neuron firings inside deep neural networks in both the NLP and computer vision domains.”
MethodEfficient Contextualized Representation: Language Model Pruning for Sequence Labeling
Liu, L., Ren, X., Shang, J., Gu, X., Peng, J., & Han, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1153 · Cited by 45 · see citing works
“Specifically, we recognize the cell states of LSTMs are more like to be the sentence embedding (Radford et al., 2017), since they are only passed to the next time stamps.”
BackgroundThe emergence of number and syntax units in LSTM language modelsPreprint
Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., & Baroni, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.07435 · Cited by 53 · see citing works
“Radford et al. (2017), for example, detected a “sentiment” grandmother cell in a language-model-trained network.”
Ammar, S., Bouwmans, T., Zaghden, N., & Neji, M. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.0769 · Cited by 43 · see citing works
Meisheri, H., & Dey, L. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1043 · Cited by 40 · see citing works
“We use SentiNeuron feature (Radford et al., 2017) from a model which is trained over 82 million Amazon review dataset.”
MethodUnsupervised Opinion Summarization with Content Planning
Amplayo, R. K., Angelidis, S., & Lapata, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17481 · Cited by 5 · see citing works
Monophonic Music Generation With a Given Emotion Using Conditional Variational Autoencoder
Grekow, J., & Dimitrova-Grekow, T. (2021) · IEEE Access · doi:10.1109/access.2021.3113829 · Cited by 39 · see citing works
“It developed the method used for generating textual product reviews with a sentiment [33] by using a single-layer multiplicative long short-term memory (mLSTM) network.”
MethodProceedings of the Fourth Arabic Natural Language Processing Workshop
El-Hajj, W., Belguith, L. H., Bougares, F., Magdy, W., Zitouni, I., Tomeh, N., & El Hajj, M. (2019) · doi:10.18653/v1/w19-46 · Cited by 37 · see citing works
- Authors
- Radford, A., Jozefowicz, R., & Sutskever, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.01444
- OpenAlex
- W2606347107
- Cited by
- 330
- References
- 47
- Citations by year
- 2025: 3 · 2024: 11 · 2023: 16 · 2022: 16 · 2021: 52 · 2020: 58 · 2019: 84 · 2018: 63 · 2017: 24 · 2016: 2 · 2013: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- sentiment analysis, unsupervised feature learning, disentangled representations, generative text, data efficiency, Stanford Sentiment Treebank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
