The DOI matches its DataCite record; 44 of the 47 references it cites were found in a registry or an index.
Learning to Generate Reviews and Discovering Sentiment
Open accessAbstract
We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Such multi-task objectives may be more effective than single-task ones (e.g., ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space (Chen et al., 2020;He et al., 2020;Radford et al., 2017;2019).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Radford et al., 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Image classifiers learn a broadly useful hierarchy of feature detectors re-representing raw pixels as edges, textures, and objects (Zeiler & Fergus, 2014).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · cited 1×
“On the MR (Pang & Lee, 2005) and CR (Hu & Liu, 2004) sentiment analysis datasets we improve the state of the art by a significant margin.”
MethodExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Research works printed but not linked: 11
- Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.
- Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin. Reexamining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 182-190. Association for Computational Linguistics, 2012.
- Munkhdalai, Tsendsuren and Yu, Hong. Neural semantic encoders. arXiv preprint arXiv:1607.04315 , 2016. · arXiv
- Ng, Andrew Y. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twentyfirst international conference on Machine learning , pp. 78. ACM, 2004.
- Pang, Bo and Lee, Lillian. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics , pp. 271. Association for Computational Linguistics, 2004.
- Wang, Sida and Manning, Christopher D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2 , pp. 90-94. Association for Computational Linguistics, 2012.
- Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems , pp. 649-657, 2015.
- Zhao, Han, Lu, Zhengdong, and Poupart, Pascal. Selfadaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070 , 2015. · arXiv
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Skip-thought vectors [26] and Byte mLSTM [27] are alternatives which go beyond word prediction with a Recurrent Neural Network, and use maximum likelihood over sequences of observations.”
BackgroundMethodUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
“…modeling can be seen as the ideal source task and a counter-part of ImageNet for NLP: It captures many facets of language relevant for downstream tasks, such as long-term dependencies (Linzen et al., 2016), hierarchical relations (Gulordava et al., 2018), and sentiment (Radford et al., 2017).”
BackgroundPei, K., Cao, Y., Yang, J., & Jana, S. (2017) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3132747.3132785 · Cited by 1,358
“It has recently been shown that each neuron in a DNN tends to independently extract a specific feature of the input instead of collaborating with other neurons for feature extraction [58, 87].”
BackgroundUnified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Metadata
- Authors
- Radford, A., Jozefowicz, R., & Sutskever, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.01444
- OpenAlex
- W2606347107
- Cited by
- 330
- References
- 47
- Citations by year
- 2025: 3 · 2024: 11 · 2023: 16 · 2022: 16 · 2021: 52 · 2020: 58 · 2019: 84 · 2018: 63 · 2017: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- sentiment analysis, unsupervised feature learning, disentangled representations, generative text, data efficiency, Stanford Sentiment Treebank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
