The DOI matches its DataCite record; 44 of the 47 references it cites were found in a registry or an index.
Learning to Generate Reviews and Discovering Sentiment
Open accessAbstract
We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Such multi-task objectives may be more effective than single-task ones (e.g., ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space (Chen et al., 2020;He et al., 2020;Radford et al., 2017;2019).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Radford et al., 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Image classifiers learn a broadly useful hierarchy of feature detectors re-representing raw pixels as edges, textures, and objects (Zeiler & Fergus, 2014).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · cited 1×
“On the MR (Pang & Lee, 2005) and CR (Hu & Liu, 2004) sentiment analysis datasets we improve the state of the art by a significant margin.”
MethodExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Research works printed but not linked: 11
- Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.
- Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin. Reexamining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 182-190. Association for Computational Linguistics, 2012.
- Munkhdalai, Tsendsuren and Yu, Hong. Neural semantic encoders. arXiv preprint arXiv:1607.04315 , 2016. · arXiv
- Ng, Andrew Y. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twentyfirst international conference on Machine learning , pp. 78. ACM, 2004.
- Pang, Bo and Lee, Lillian. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics , pp. 271. Association for Computational Linguistics, 2004.
- Wang, Sida and Manning, Christopher D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2 , pp. 90-94. Association for Computational Linguistics, 2012.
- Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems , pp. 649-657, 2015.
- Zhao, Han, Lu, Zhengdong, and Poupart, Pascal. Selfadaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070 , 2015. · arXiv
Cited by
Sorted by citations
Items 301–339 of 339
Towards Controlled Transformation of Sentiment in SentencesPreprint
Leeftink, W., & Spanakis, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11467 · Cited by 2
AspeRa: Aspect-based Rating Prediction ModelPreprint
Nikolenko, S., Tutubalina, E. V., Malykh, V. A., Shenbin, I., & Alekseev, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.07829 · Cited by 1
“Our results can also be viewed as part of the research effort to analyze and interpret deep neural networks, a very important recent trend [12,30].”
Towards Controllable and Personalized Review GenerationPreprint
Li, P., & Tuzhilin, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03506 · Cited by 1
“…reviews of good quality, including Aspect-Aware Representations (Ni and McAuley, 2018), Gated Contexts to Sequences(Tang et al., 2016), RNN(Yao et al., 2017), Aspect-Sentiment Score(Zang and Wan, 2017), Generative Concatenative Nets(Lipton et al., 2015) and Sentiment Units(Radford et al., 2017).”
BackgroundEmergent Properties of Finetuned Language Representation ModelsPreprint
Matton, A., & de Oliveira, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10832 · Cited by 3
“We first begin by investigating, in a similar spirit to Radford et al. (2017), which dimensions or neurons from the pretrained BERT model directly exhibit useful information for our three datasets.”
BackgroundHow Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning AgentsPreprint
Meyes, R., Schneider, M., & Meisen, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03237 · Cited by 2
“Similar efforts are made in the field of natural language processing (NLP), where recurrent neural networks (RNNs) are investigated for their representations of linguistic properties, contextual understanding or sentiment [14,15,16,17].”
BackgroundAssessing the Helpfulness of Review Content for Explaining RecommendationsPreprint
Hernandez-Bocanegra, D. C., & Ziegler, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06328 · Cited by 1
“We plan to base our work on developments from argument mining [7, 8, 18] (for the supervised detection of units and its relationships on a multi-sentence level), sentiment and subjectivity analysis (for detection of objective/subjective units on a sentence level [16]).”
MethodMusic Classification in MIDI Format based on LSTM MdelPreprint
Xia, Y., Jiang, Y., & Ye, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07739 · Cited by 1
“[8] Radford A, Jozefowicz R, Sutskever I.”
SAM: Semantic Attribute Modulation for Language Modeling and Style VariationPreprint
Hu, W., Hua, L., Li, L., Su, H., Wang, T., Chen, N., & Zhang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00117
Epilogue: Frontiers of NLP in the Deep Learning Era
Deng, L., & Liu, Y. (2018) · Springer eBooks · doi:10.1007/978-981-10-5209-5_11 · Cited by 2
Understanding the Importance of Single Directions via Representative SubstitutionPreprint
Chen, L., Ding, H., Li, Q., Li, Z., Peng, J., & Li, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05586
Towards Language Agnostic Universal RepresentationsPreprint
Aghajanyan, A., Song, X., & Tiwary, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08510
“Even further, Radford et al. (2017) show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured…”
BackgroundInterpreting author intentions by analyzing story modulation
Bandler, S. C. (2019) · DSpace@MIT (Massachusetts Institute of Technology)
Adaptive Noise Injection: A Structure-Expanding Regularization for RNNPreprint
Li, R., Shuang, K., Gu, M., & Su, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.10885
“Correspondence to: Rui Li <lirui@bupt.edu.cn>. ation (Radford et al., 2017; Merity et al., 2017).”
Sparsity Emerges Naturally in Neural Language ModelsPreprint
Saphra, N., & Lopez, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.01817
“, 2015), or properties like sentiment (Radford et al., 2017).”
BackgroundSemi-Supervised Self-Growing Generative Adversarial Networks for Image RecognitionPreprint
Wang, H., Xu, Z., Xu, J., An, W., Lei Zhang, A., & Dai, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03850
“[46] trained an mLSTM RNN on Amazon reviews to learn a language model and then used its internal cell state from the last time step as features for the subsequent supervised task of sentiment analysis of Amazon reviews.”
MethodNeural architectures for natural language understanding
Tay, Y. (2019) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/83291
Affect inTweets: A Transfer Learning Approach.
Zhang, L., Huang, H.-L., Yu, Y., & MOLDOVAN, D. I. (2020) · Language Resources and Evaluation
Linking average- and worst-case perturbation robustness via class selectivity and dimensionalityPreprint
Leavitt, M. L., & Morcos, A. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07693
“Evidence regarding the importance of selectivity has mostly relied on single unit ablation, and has been equivocal (Radford et al., 2017; Morcos et al., 2018; Amjad et al., 2018; Zhou et al., 2018; Donnelly and Roegiest, 2019; Dalvi et al., 2019a).”
BackgroundLow-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural NetworksPreprint
Fanthomme, A., & Monasson, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10435
“…studies have focused on the emergence of disentangled representations, which have been shown to be relevant in both Natural Language Processing (Radford et al., 2017) and Computer Vision (Denton and Birodkar, 2017; Lee et al., 2018), suggesting that the optimal type of representation might…”
BackgroundExploring Deep Neural Networks and Transfer Learning for Analyzing Emotions in TweetsPreprint
Senarath, Y., & Thayasivam, U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06025
“Early examples of research into sentiment analysis involve in polarity classification of the textual input (Pang et al., 2002; Tang et al., 2014; Dong et al., 2014; Radford et al., 2017).”
BackgroundMining Program Properties From Neural Networks Trained on Source Code EmbeddingsPreprint
Saletta, M., & Ferretti, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05442
“authors of [4] and [13] did for images and natural language, respectively.”
BackgroundMethodInfluentialSelf-supervised Learning for Semantic Sentence Matching with Dense Transformer Inference Network
Yu, F., Wang, J., Tao, D., Cheng, N., & Xiao, J. Z. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85896-4_21
A Behavior-Driven Forum Spammer Recognition Method with Its Application in Automobile Forums
Su, H., Ren, M., Wang, A., Tang, X., Ni, X., & Zhao, F. (2021) · Mathematical Problems in Engineering · doi:10.1155/2021/7682579
“[20] constructed an unsupervised representation learning system, which reached an accuracy of 91.”
BackgroundSubjective Prediction of Questions in Q & A System based on the Open Domain of Daily Life
Wang, W., Yue, Y., & Zhu, X. (2022) · International Conference on Control Engineering and Artificial Intelligence (CCEAI) · doi:10.1145/3522749.3523085
“[14] adopted a pre-trained LSTM language model.”
Forensic Suicidal Inquiry of Depressed Individuals using LSTM and Convolutional Neural Networks
(2019) · International Journal of Innovative Technology and Exploring Engineering · doi:10.35940/ijitee.f1020.0486s419
Automatic Generation of News Commentary on Social Media Based on Self‐Attention Mechanism
Qu, M. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/2374590
“In [8], authors have proposed a model for generating product reviews based on attribute information such as user, product, and rating which were proposed.”
BackgroundCan we Debunk Disinformation by Leveraging SpeakerCredibility and Perplexity Measures?Preprint
Khilji, A. F. U. R., Sachan, A., Lachi, D., Singh, A. V., & Singh, T. D. (2023) · Research Square · doi:10.21203/rs.3.rs-2764182/v1
Innovative procedure for soppiness investigation spending writing and facemask gratitude
Kanbar, W., & Hdedeh, S. (2021) · International Journal of Cloud Computing and Database Management · doi:10.33545/27075907.2021.v2.i1a.24
Computer-assisted detection of typologically relevant semantic shifts in world languages
Gruntov, I., & Elisei, R. (2023) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2023-22-161-171
Composing music based on machine learning
Zhang, J. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/48/20241494
Translating and Optimising Computational Microscopy Algorithms with Large Language Models
Guzzi, F., Kourousias, G., Pugliese, R., Gianoncelli, A., & Billè, F. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569791
“To gain this ability, Generative Pre-Trained Transformers (GPTs), are trained in a self-supervised learning approach [13] on a large corpus of text: while predicting the next most probable token (a sequence of characters), grammar and semantics rules of the dataset [14] are inferred.”
BackgroundZhang, Y., Niu, L., Wang, X., Du, D., & Zhang, Z. (2025) · Computers & Geosciences · doi:10.1016/j.cageo.2025.105871
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
“…language tasks, one can achieve state of the art results as shown in recent work (Devlin et al., 2018; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; 2017; Ramachandran et al., 2016; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019).”
BackgroundResultLearning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Subramanian, S., Trischler, A., Bengio, Y., & Pal, C. J. (2018) · arXiv · doi:10.48550/arxiv.1804.00079 · Cited by 207
“Shi et al. (2016) also present evidence that sequence-to-sequence parsers (Vinyals et al.”
BackgroundMethodUnsupervised representation learning for time series with temporal neighborhood coding
Tonekaboni, S., Eytan, D., & Goldenberg, A. (2021) · arXiv · doi:10.48550/arxiv.2106.00750 · Cited by 170
“…learning is well studied in domains such as vision (Donahue & Simonyan, 2019; Denton et al., 2017; Radford et al., 2015) and natural language processing (Radford et al., 2017; Young et al., 2018; Mikolov et al., 2013), but has been underexplored in the literature for time series settings.”
BackgroundMulti-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
Tripathi, S., Tripathi, S., & Beigi, H. (2018) · arXiv · doi:10.48550/arxiv.1804.05788 · Cited by 132
Metadata
- Authors
- Radford, A., Jozefowicz, R., & Sutskever, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.01444
- OpenAlex
- W2606347107
- Cited by
- 330
- References
- 47
- Citations by year
- 2025: 3 · 2024: 11 · 2023: 16 · 2022: 16 · 2021: 52 · 2020: 58 · 2019: 84 · 2018: 63 · 2017: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- sentiment analysis, unsupervised feature learning, disentangled representations, generative text, data efficiency, Stanford Sentiment Treebank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
