The DOI matches its DataCite record; 44 of the 47 references it cites were found in a registry or an index.
Learning to Generate Reviews and Discovering Sentiment
Open accessAbstract
We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Such multi-task objectives may be more effective than single-task ones (e.g., ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space (Chen et al., 2020;He et al., 2020;Radford et al., 2017;2019).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Radford et al., 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Image classifiers learn a broadly useful hierarchy of feature detectors re-representing raw pixels as edges, textures, and objects (Zeiler & Fergus, 2014).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · cited 1×
“On the MR (Pang & Lee, 2005) and CR (Hu & Liu, 2004) sentiment analysis datasets we improve the state of the art by a significant margin.”
MethodExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Research works printed but not linked: 11
- Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.
- Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin. Reexamining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 182-190. Association for Computational Linguistics, 2012.
- Munkhdalai, Tsendsuren and Yu, Hong. Neural semantic encoders. arXiv preprint arXiv:1607.04315 , 2016. · arXiv
- Ng, Andrew Y. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twentyfirst international conference on Machine learning , pp. 78. ACM, 2004.
- Pang, Bo and Lee, Lillian. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics , pp. 271. Association for Computational Linguistics, 2004.
- Wang, Sida and Manning, Christopher D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2 , pp. 90-94. Association for Computational Linguistics, 2012.
- Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems , pp. 649-657, 2015.
- Zhao, Han, Lu, Zhengdong, and Poupart, Pascal. Selfadaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070 , 2015. · arXiv
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Skip-thought vectors [26] and Byte mLSTM [27] are alternatives which go beyond word prediction with a Recurrent Neural Network, and use maximum likelihood over sequences of observations.”
BackgroundMethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
“…modeling can be seen as the ideal source task and a counter-part of ImageNet for NLP: It captures many facets of language relevant for downstream tasks, such as long-term dependencies (Linzen et al., 2016), hierarchical relations (Gulordava et al., 2018), and sentiment (Radford et al., 2017).”
BackgroundPei, K., Cao, Y., Yang, J., & Jana, S. (2017) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3132747.3132785 · Cited by 1,358
“It has recently been shown that each neuron in a DNN tends to independently extract a specific feature of the input instead of collaborating with other neurons for feature extraction [58, 87].”
BackgroundUnified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Contextual String Embeddings for Sequence Labeling
Akbik, A., Blythe, D. A. J., & Vollgraf, R. (2018) · International Conference on Computational Linguistics · Cited by 1,086
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“This success has been fueled by pretraining on large quantities of raw text using pretext objectives, such as language modeling (Radford et al., 2017) or word vectors (Devlin et al., 2019), that require no supervision.”
BackgroundThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
SemEval-2018 Task 1: Affect in Tweets
Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1001 · Cited by 746
“• Sentiment neurons (Radford et al., 2017): a byte-level recurrent language model for learning sentence representations.”
BackgroundFelbo, B., Mislove, A. E., Søgaard, A., Rahwan, I., & Lehmann, S. (2017) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 712
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572
“…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
InfluentialLow-N protein engineering with data-efficient deep learning
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., & Church, G. M. (2021) · Nature Methods · doi:10.1038/s41592-021-01100-y · Cited by 422
Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation ModelsPreprint
Guimaraes, G. L., Sánchez-Lengeling, B., Outeiral, C., Farias, P. L. C., & Aspuru‐Guzik, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10843 · Cited by 702
“In natural language processing, a given sentiment might be emphasized, maybe for producing movie reviews [Radford et al., 2017].”
BackgroundFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
“Since the reward model needs to understand language, we initialize it as a random linear function of the final embedding output of the language model policy ρ following Radford et al. (2018) (see section 4.”
BackgroundMethodSelf-citationUnderstanding the role of individual units in a deep neural network
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., & Torralba, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907375117 · Cited by 375
“Intriguingly, within state-of-the-art deep networks, it has been observed that many single units match human-interpretable concepts that were not explicitly taught to the network: units have been found to detect objects, parts, textures, tense, gender, context, and sentiment (1–7).”
BackgroundAnalysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403
“PDA [84] image feature importance local, ante-hoc DVE [83] image visualization local, post-hoc Grad-CAM [76] image saliency mask local, post-hoc Grad-CAM++ [78] image saliency mask local, post-hoc Smooth-Grad [86] image sensitivity analysis local, ante-hoc ME [88] image visualization local, post-hoc Summit [89] image…”
MethodA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. F. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 362
“, 2017), GANs (Wang & Wan, 2018) or Seq2Seq models (Radford et al., 2017).”
Fine-tuned Language Models for Text Classification.Preprint
Howard, J., & Ruder, S. (2018) · arXiv (Cornell University) · Cited by 357
Deep Semantic Feature Learning for Software Defect Prediction
Wang, S., Liu, T., Nam, J., & Tan, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877612 · Cited by 331
“[79] focused on understanding the properties of representations learned by byte-level recurrent language models for sentiment analysis.”
BackgroundWieting, J., & Gimpel, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1042 · Cited by 356
Exposing the Fingerprint: Dissecting the Impact of the Wireless Channel on Radio Fingerprinting
Al-Shawabka, A., Restuccia, F., D’Oro, S., Jian, T., Rendon, B. C., Soltani, N., Dy, J., Ioannidis, S., Chowdhury, K. R., & Melodia, T. (2020) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom41043.2020.9155259 · Cited by 312
“…in more mature and widely-explored learning domains such as computer vision and natural language processing, where massive-scale labeled datasets such as MNIST for image classification [20] and IMDb for sentiment analysis [21] have been completely available to the research community for many years.”
BackgroundChallenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288
“research on modeling empathetic interactions in conversation [100]. We will present dialog datasets in Section7. Controlling the emotion or sentiment has become a popular topic in language generation [34, 45, 97]. In [97], an RNN-based language model is trained on large-scale review data where some neurons are reporte…”
MethodZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020) · Distill · doi:10.23915/distill.00024.001 · Cited by 288
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286
Controlling Linguistic Style Aspects in Neural Language Generation
Ficler, J., & Goldberg, Y. (2017) · arXiv.org · doi:10.18653/v1/w17-4912 · Cited by 334
“…the content of the generated text (Lipton et al., 2015; Kid-don et al., 2016; Lebret et al., 2016; Kiddon et al., 2016; Tang et al., 2016; Radford et al., 2017), while a few model more stylistic aspects of the generated text such as the identity of the speaker in a dialog setting (Li et…”
BackgroundUnsupervised Pretraining for Sequence to Sequence Learning
Ramachandran, P., Liu, P., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1039 · Cited by 251
“Dai and Le (2015); Radford et al. (2017) are amongst the rare studies which showed the benefits of pretraining in a semi-supervised learning setting.”
BackgroundA Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 290
“…interesting tasks related to NLP like: text games [222], language to program execution [223], extraction of information [224], sentiment analysis [225], knowledge graph reasoning [226], automatic query reformulation [227], information retrival [228], summarization [229], [230] and semantic…”
BackgroundEvaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248
“LLM representations have also been observed to encode emotion and sentiment [36, 64].”
BackgroundSentiment Analysis for Fake News Detection
Alonso, M. Á., Vilares, D., Gómez‐Rodríguez, C., & Vilares, J. (2021) · Electronics · doi:10.3390/electronics10111348 · Cited by 267
“For instance, Radford et al. [141] showed how pre-training an LSTM on a large data set of Amazon reviews at the character level (byte level) could obtain state-of-the-art results for SA on standard data sets by simply fine-tuning on a few supervised examples.”
BackgroundPretraining-Based Natural Language Generation for Text Summarization
Zhang, H., Cai, J., Xu, J., & Wang, J. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1074 · Cited by 267
Deep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230
“1) Sentiment Analysis [93]–[95]: The input is a sequence of words, and the output is a single sentiment (positive, negative, or neutral).”
BackgroundPersuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222
“tive LSTM (mLSTM) network on 80 million Amazon product reviews to extract 4096-dimension character-level features (Radford et al., 2017)3.”
BackgroundMethodGeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220
“Natural language generation has seen great progress with the advent of Transformers (Vaswani et al., 2017) and large scale training (Radford et al., 2017, 2018, 2019; Brown et al., 2020).”
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
“Finally, it is worth mentioning that [81] presents the discovery that a single neuron unit of a DNN can perform alone a sentiment analysis task after the training of the network reaching the same level of performance of strong baselines.”
BackgroundEmotion Recognition on Twitter: Comparative Study and Training a Unison Model
Colnerič, N., & Demšar, J. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2807817 · Cited by 187
“Radford et al. [14] even reported on discovering the sentiment unit while training to generate reviews.”
BackgroundFrom attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181
ArtEmis: Affective Language for Visual Art
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01140 · Cited by 161
“Radford and colleagues [49] discovered that a single unit in recurrent language models trained without sentiment labels, is automatically learning concepts of sentiment; enabling sentiment-oriented manipulation by fixing the sign of that unit.”
BackgroundCompiler fuzzing through deep learning
Cummins, C., Petoumenos, P., Murray, A. C., & Leather, H. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213848 · Cited by 163
“Our approach is inspired by breakthrough results in modeling challenging and high dimensional datasets through unsupervised learning [4, 27, 28].”
MethodLearning to Compose Task-Specific Tree Structures
Choi, J., Yoo, K. M., & Lee, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11975 · Cited by 25
“…NSE (Munkhdalai and Yu, 2017a) 89.7 52.8 CT-LSTM (single) (Looks et al., 2017) 89.4 52.3 CT-LSTM (ensemble) (Looks et al., 2017) 90.2 53.6 byte-mLSTM (Radford et al., 2017) 91.8 52.9 BCN+Char+CoVe (McCann et al., 2017) 90.3 53.7 The result shows that our model effectively maps similar sentences…”
InfluentialInductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT
Li, X., & Fourches, D. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-00430-x · Cited by 150
Canine : Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00448 · Cited by 205
On Completeness-aware Concept-Based Explanations in Deep Neural NetworksPreprint
Yeh, C., Kim, B., Arık, S. Ö., Li, C., Pfister, T., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07969 · Cited by 144
“Some use dimensionality reduction methods [34, 35], while others uncover higher level human-relatable concepts by dimensionality reduction (e.g. for speech [36] and language [37, 38]).”
Sekaran, K., Chandana, P., Krishna, N. M., & Kadry, S. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7419-5 · Cited by 133
“The Results of the prediction of cancer with respect to the threshold parameters fed to the Deep learning is presented by identifying the tumor [14] rate of spread in the head section of the pancreas, after diagnosing and treatment, the spread of tumor size has been decreased, which helped patient to get cured than th…”
BackgroundWhat Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., & Glass, J. R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016309 · Cited by 18
“Some recent work on neural language models and machine translation analyzes specific neurons of length (Qian, Qiu, and Huang 2016a; Shi, Knight, and Yuret 2016) and sentiment (Radford, Jozefowicz, and Sutskever 2017).”
BackgroundExplainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
Lin, Y., Ren, P., Chen, Z., Ren, Z., Ma, J., & de Rijke, M. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2906190 · Cited by 134
An Analysis of Neural Language Modeling at Multiple ScalesPreprint
Merity, S., Keskar, N. S., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08240 · Cited by 187
“…LMs are useful in many applications including speech recognition (Yu & Deng, 2014), machine translation (Koehn, 2009), natural language generation (Radford et al., 2017; Merity et al., 2017a), learning token embeddings, and as a general-purpose feature extractor for downstream tasks (P. Matthew,…”
BackgroundLAMOL: LAnguage MOdeling for Lifelong Language LearningPreprint
Sun, F.-K., Ho, C.-H., & Lee, H.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03329 · Cited by 201
Learning Language Representations for Typology Prediction
Malaviya, C., Neubig, G., & Littell, P. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1268 · Cited by 121
“sentiment (Radford et al., 2017), we expect that the cell states will represent features that may be”
BackgroundIdentifying and Controlling Important Neurons in Neural Machine TranslationPreprint
Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01157 · Cited by 162
“In contrast, previous work found meaningful individual neurons in computer vision (Zeiler & Fergus, 2014; Zhou et al., 2016; Bau et al., 2017, among others) and in a few NLP tasks (Karpathy et al., 2015; Radford et al., 2017; Qian et al., 2016a).”
BackgroundResult(2019) · doi:10.18653/v1/n19-1002 · Cited by 118
“Radford et al. (2017), for example, detected a “sentiment” grandmother cell in a language-model-trained network.”
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Lu, L., Murray, I., & Renals, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07959 · Cited by 170
SEDAT: Sentiment and Emotion Detection in Arabic Text Using CNN-LSTM Deep Learning
Abdullah, M., Hadzikadicy, M., & Shaikhz, S. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00134 · Cited by 112
“In this system, we uses word and document embedding models with feature vectors extracted from Arabic and translated tweets by using the AffectiveTweets package, Deepmoji, and Unsupervised Sentiment Neurons.”
MethodEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
“Radford et al. (2017, 2019) and Brown et al. (2020) demonstrated that language models trained with an unsupervised objective can perform a variety of tasks out-of-the-box.”
BackgroundAdelani, D. I., Mai, H., Fang, F., Nguyen, H. H., Yamagishi, J., & Echizen, I. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-44041-1_114 · Cited by 121
“further propose a “skill-up” method in which an LM is explicitly conditioned by a specified sentiment. This method requires a natural language processing expert to train a tailored LM. Radford et al. [31] reported that a sentiment neuron can be learned by using a single-layer multiplicative LSTM (mLSTM) [28]. The senti…”
MethodXia, Y., Xiao, Z., Jazdi, N., & Weyrich, M. (2024) · IEEE Access · doi:10.1109/access.2024.3415470 · Cited by 119
“The proposed ‘‘semantic node’’ is specified in Table 2 and visualized in Figure 10 .”
BackgroundGenerating and designing DNA with deep generative modelsPreprint
Killoran, N., Lee, L. J., Delong, A., Duvenaud, D., & Frey, B. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06148 · Cited by 166
“[35, 36] demonstrated that one can sometimes inspect the activations of RNNs trained in an unsupervised manner and find neuron units whose activation captures some property of interest in the sequence.”
BackgroundVariational Pretraining for Semi-supervised Text Classification
Gururangan, S., Dang, T., Card, D., & Smith, N. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1590 · Cited by 108
“, 2000), but larger and more reliable gains have been obtained by using contextual embeddings trained with a language modeling (LM) objective on massive amounts of text from domains such as Wikipedia or news (Peters et al., 2018a; Devlin et al., 2019; Radford et al., 2018; Howard and Ruder, 2018).”
BackgroundPrivFT: Private and Fast Text Classification With Homomorphic Encryption
Badawi, A. A., Hoang, L., Mun, C. F., Laine, K., & Aung, K. M. M. (2020) · IEEE Access · doi:10.1109/access.2020.3045465 · Cited by 103
“Much of the recent NLP research has focused on transfer learning techniques such as pre-training word embeddings [37], [38], or pre-training language models on larger datasets and fine-tuning them for task-specific learning [5]–[8].”
BackgroundMethodxLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299
Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline
Ethayarajh, K. (2018) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w18-3012 · Cited by 93
Language Models as Agent Models
Andreas, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.423 · Cited by 92
Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
Shi, W., Wang, X., Oh, Y. J., Zhang, J., Sahay, S., & Yu, Z. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376843 · Cited by 93
“Character information was calculated by a pretrained char embedding [38].”
MethodBaşarslan, M. S., & Kayaalp, F. (2023) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-022-00386-3 · Cited by 88
Extractive Opinion Summarization in Quantized Transformer Spaces
Angelidis, S., Amplayo, R. K., Suhara, Y., Wang, X., & Lapata, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00366 · Cited by 85
“We use a Centroid method that encodes the entity’s reviews with BERT (average token vector; Devlin et al., 2019) or SentiNeu-ron (Radford et al., 2017), and picks the one closest to the mean review vector.”
MethodAlonso, M. Á., Vilares, D., Gómez‐Rodríguez, C., & Vilares, J. (2013) · Electronics · doi:10.3390/electronics · Cited by 82
Berge, G. T., Granmo, O., Tveit, T. O., Goodwin, M., Jiao, L., & Matheussen, B. V. (2019) · IEEE Access · doi:10.1109/access.2019.2935416 · Cited by 86
“…adversarial training (S. Wang and Manning 2012; Mesnil et al. 2014; Zhao 2016; Johnson and Zhang 2015; Dieng et al. 2016; Miyato, Dai, and Goodfellow 2016; Sachan, Zaheer, and Salakhutdinov 2018; Radford, Jozefowicz, and Sutskever 2017; Gray, Radford, and Kingma 2017; Howard and Ruder 2018).”
BackgroundAugmenting Neural Networks with First-order Logic
Li, T., & Srikumar, V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1028 · Cited by 87
“Node semantics may be assigned during model design (e.g. attention), or incidentally discovered in post hoc analysis (e.g., Le et al., 2012; Radford et al., 2017, and others).”
Do Convolutional Networks need to be Deep for Text Classification ?Preprint
Le, H. T., Cerisara, C., & Denis, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04108 · Cited by 92
Controllable Text Generation.Preprint
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017) · arXiv (Cornell University) · Cited by 75
Anomaly Detection in Operating System Logs with Deep Learning-Based Sentiment Analysis
Studiawan, H., Sohel, F. A., & Payne, C. (2020) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2020.3037903 · Cited by 69
Protein design via deep learning
Ding, W., Nakai, K., & Gong, H. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac102 · Cited by 66
“Although the data mining ability of RNN architecture used by UniRep might be inferior compared with current popular ones in the field of nature language processing like transformer, the basic conceptions it came up with and the impressive extensions it showed still influenced following researches a lot.”
InfluentialDeep Learning for Cognitive Neuroscience
Storrs, K. R., & Kriegeskorte, N. (2020) · The MIT Press eBooks · doi:10.7551/mitpress/11442.003.0077 · Cited by 74
hULMonA: The Universal Language Model in Arabic
Jundi, O. E., Antoun, W., Droubi, N. E., Hajj, H. M., El-Hajj, W., & Shaban, K. B. (2019) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/w19-4608 · Cited by 77
“…to obtain general understanding of a particular language due to its ability of capturing many aspects of language relevant for downstream tasks, such as long-term dependencies (Linzen et al., 2016), hierarchical relations (Gulordava et al., 2018), and sentiment orientation (Radford et al., 2017).”
BackgroundUnsupervised Opinion Summarization with Noising and Denoising
Amplayo, R. K., & Lapata, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.175 · Cited by 68
“, 2017), a centroidbased method which represents the input as IDFweighted word embeddings and selects as summary the review closest to the centroid, and (c) SENTINEURON, which is similar to WORD2VEC but uses a language model called Sentiment Neuron (Radford et al., 2017) as input representation.”
MethodDeep Learning for Software Defect Prediction
Omri, S., & Sinz, C. (2020) · doi:10.1145/3387940.3391463 · Cited by 62
“Radford et al. [42] focus on understanding the properties of representations learned by byte-level recurrent language models for sentiment analysis.”
BackgroundProceedings of the First Workshop on Abusive Language Online
(2017) · doi:10.18653/v1/w17-30 · Cited by 63
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1797 · Cited by 117
“Therefore, we alternate between supervised training on the QA pairs and pretraining on Open Web Text (Radford et al., 2017).”
MethodDeep learning frameworks for protein–protein interaction prediction
Hu, X., Feng, C., Ling, T., & Chen, M. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.06.025 · Cited by 61
“The architectures of UniRep contain a 1,900-dimensional single-layer multiplicative long short-term-memory (LSTM) recurrent neural networks (RNNs) [59], a four-layer stacked multiplicative LSTM of 256 dimensions and a four-layer stacked multiplicative of 64 dimensions.”
MethodCharacter-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56
“Character-level modeling has shown promise in many areas such as sentiment analysis [35], question answering [21] and classification [50], and is an exciting area due to its simplicity and the ability to easily adapt to other languages.”
BackgroundFleming, R. W., & Storrs, K. R. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.07.004 · Cited by 54
Story Centaur: Large Language Model Few Shot Learning as a Creative Writing Tool
Swanson, B., Mathewson, K. W., Pietrzak, B., Chen, S., & Dinalescu, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.29 · Cited by 59
“While most work concerned itself primarily with topping each other’s GLUE and SUPERGLUE scores (Wang et al., 2019), work from OpenAI kept the torch lit for investigating the emergent abilities of representation learning (Radford et al., 2017, 2018, 2019; Brown et al., 2020).”
BackgroundDeep Learning-Based Sentiment Classification: A Comparative Survey
Mabrouk, A., Redondo, R. P. D., & Kayed, M. (2020) · IEEE Access · doi:10.1109/access.2020.2992013 · Cited by 63
“Word embedding handles many SA tasks such as aspect term extraction [64], SC [32], [59], [65], [66], among other.”
BackgroundMethodInfluentialSimilarity Analysis of Contextual Word Representation Models
Wu, J., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.422 · Cited by 69
“While neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons (Qian et al., 2016; Shi et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Bau et al., 2019; Dalvi et al., 2019; Baan et al., 2019).”
BackgroundPinheiro, L. D. S., & Dras, M. (2017) · Cited by 54
SeerNet at SemEval-2018 Task 1: Domain Adaptation for Affect in Tweets
Duppada, V., Jain, R., & Hiray, S. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1002 · Cited by 51
“(Radford et al., 2017) developed an unsupervised system which learned an excellent representation of sentiment.”
BackgroundUnderstanding Pre-trained BERT for Aspect-based Sentiment Analysis
Xu, H., Shu, L., Yu, P., & Liu, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.21 · Cited by 57
“This is in contrast with the findings of sentiment neuron in pre-trained casual LMs (Radford et al., 2017) and we suspect the small hidden space (768) does not allow such a single neuron to exist.”
ResultInfluentialElhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.10652 · Cited by 51
Technology Solutions to Combat Online Harassment
Kennedy, G., McCollough, A. W., Dixon, E., Bastidas, A., Ryan, J. B., Loo, C., & Sahay, S. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3011 · Cited by 51
“(Radford et al., 2017) and have the potential to resist the ”adversarial” tokens described in (Hosseini et al., 2017).”
BackgroundStojanovski, D., Strezoski, G., Madjarov, G., Dimitrovski, I., & Chorbev, I. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6168-1 · Cited by 52
“[38] showed that training a language model in an unsupervised manner can also be used for sentiment analysis.”
BackgroundChemlistem: chemical named entity recognition using recurrent neural networks
Corbett, P. T., & Boyle, J. (2018) · Journal of Cheminformatics · doi:10.1186/s13321-018-0313-8 · Cited by 49
“Radford et al. [16] trained a byte-level language model on product reviews, and were able to use this model to train a sentiment analysis system with high data efficiency.”
MethodYou, Q., Jin, H., & Luo, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10121 · Cited by 63
“Recently, a sentiment unit is discovered in a RNN based generation model (Radford, Jozefowicz, and Sutskever 2017).”
BackgroundEcco: An Open Source Library for the Explainability of Transformer Language Models
Alammar, J. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.30 · Cited by 53
“Previous work (Karpathy et al., 2015; Strobelt et al., 2017; Poerner et al., 2018; Radford et al., 2017; Olah et al., 2017, 2018; Bau et al., 2018; Dalvi et al., 2019; Rethmeier et al., 2020) has examined neuron firings inside deep neural networks in both the NLP and computer vision domains.”
MethodEfficient Contextualized Representation: Language Model Pruning for Sequence Labeling
Liu, L., Ren, X., Shang, J., Gu, X., Peng, J., & Han, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1153 · Cited by 45
“Specifically, we recognize the cell states of LSTMs are more like to be the sentence embedding (Radford et al., 2017), since they are only passed to the next time stamps.”
BackgroundThe emergence of number and syntax units in LSTM language modelsPreprint
Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., & Baroni, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.07435 · Cited by 69
“Radford et al. (2017), for example, detected a “sentiment” grandmother cell in a language-model-trained network.”
Ammar, S., Bouwmans, T., Zaghden, N., & Neji, M. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.0769 · Cited by 43
Meisheri, H., & Dey, L. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1043 · Cited by 40
“We use SentiNeuron feature (Radford et al., 2017) from a model which is trained over 82 million Amazon review dataset.”
MethodUnsupervised Opinion Summarization with Content Planning
Amplayo, R. K., Angelidis, S., & Lapata, M. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i14.17481 · Cited by 39
Monophonic Music Generation With a Given Emotion Using Conditional Variational Autoencoder
Grekow, J., & Dimitrova-Grekow, T. (2021) · IEEE Access · doi:10.1109/access.2021.3113829 · Cited by 39
“It developed the method used for generating textual product reviews with a sentiment [33] by using a single-layer multiplicative long short-term memory (mLSTM) network.”
Method
Metadata
- Authors
- Radford, A., Jozefowicz, R., & Sutskever, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.01444
- OpenAlex
- W2606347107
- Cited by
- 330
- References
- 47
- Citations by year
- 2025: 3 · 2024: 11 · 2023: 16 · 2022: 16 · 2021: 52 · 2020: 58 · 2019: 84 · 2018: 63 · 2017: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- sentiment analysis, unsupervised feature learning, disentangled representations, generative text, data efficiency, Stanford Sentiment Treebank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
