The DOI matches its DataCite record; 44 of the 47 references it cites were found in a registry or an index.
Learning to Generate Reviews and Discovering Sentiment
Open accessAbstract
We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Such multi-task objectives may be more effective than single-task ones (e.g., ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space (Chen et al., 2020;He et al., 2020;Radford et al., 2017;2019).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Radford et al., 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Image classifiers learn a broadly useful hierarchy of feature detectors re-representing raw pixels as edges, textures, and objects (Zeiler & Fergus, 2014).”
BackgroundConvolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · cited 1×
“On the MR (Pang & Lee, 2005) and CR (Hu & Liu, 2004) sentiment analysis datasets we improve the state of the art by a significant margin.”
MethodExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“Much of the early research into modern deep learning was developed and validated via this approach (Hinton & Salakhutdinov, 2006) (Huang et al., 2007) (Vincent et al., 2008) (Coates et al., 2010) (Le, 2013).”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Research works printed but not linked: 11
- Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.
- Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan R, Zemel, Richard, Urtasun, Raquel, Torralba, Antonio, and Fidler, Sanja. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin. Reexamining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 182-190. Association for Computational Linguistics, 2012.
- Munkhdalai, Tsendsuren and Yu, Hong. Neural semantic encoders. arXiv preprint arXiv:1607.04315 , 2016. · arXiv
- Ng, Andrew Y. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twentyfirst international conference on Machine learning , pp. 78. ACM, 2004.
- Pang, Bo and Lee, Lillian. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics , pp. 271. Association for Computational Linguistics, 2004.
- Wang, Sida and Manning, Christopher D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2 , pp. 90-94. Association for Computational Linguistics, 2012.
- Zhang, Xiang, Zhao, Junbo, and LeCun, Yann. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems , pp. 649-657, 2015.
- Zhao, Han, Lu, Zhengdong, and Poupart, Pascal. Selfadaptive hierarchical sentence model. arXiv preprint arXiv:1504.05070 , 2015. · arXiv
Cited by
Sorted by citations
Items 201–300 of 339
Character Eyes: Seeing Language through Character-Level Taggers
Pinter, Y., Marone, M., & Eisenstein, J. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4811 · Cited by 8
“Radford et al. (2017) find that a character-level language model can implicitly assign a single unit to track sentiment, without being directly supervised.”
BackgroundAn Attention-based Approach to Detect Emotion from Tweets
Ahmed, S., Reyadh, A. S., Tabsun, F., Shah, F. M., & Shaafi, A. I. (2020) · International Conference of Computer and Informatics Engineering (IC2IE) · doi:10.1109/ic2ie50715.2020.9274600 · Cited by 8
“The authors have proposed a model using DeepEmoji [18], Skip Thought Vectors [19], Unsupervised Sentiment Neuron [20] making an ensemble system to detect emotions in tweets.”
MethodAuthorship Identification of Electronic Texts
Khonji, M., Iraqi, Y., & Mekouar, L. (2021) · IEEE Access · doi:10.1109/access.2021.3098192 · Cited by 8
“For example, Radford et al. found that, by training a large enough byte-level LSTM model for a month to solve the text review generation problem using a dataset comprised of millions of Amazon reviews, the model had indirectly also learned a state of art sentiment analyzer [56].”
MethodHahn, M., & Baroni, M. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00283 · Cited by 13
“Radford et al. (2017) focused on CNLMs deployed in the domain of sentiment analysis, where they found the network to specialize a unit for sentiment tracking.”
BackgroundDisentangled Representations for Manipulation of Sentiment in TextPreprint
Larsson, M., Nilsson, A., & Kågebäck, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.10066 · Cited by 8
“Recent work in text generation (Hu et al., 2017; Radford et al., 2017) has shown that it is possible to generate random sentences where the sentiment can be chosen as an input parameter.”
BackgroundBreaking the Activation Function Bottleneck through Adaptive ParameterizationPreprint
Flennerhag, S., Yin, H., Keane, J. A., & Elliot, M. J. (2018) · Research Explorer (The University of Manchester) · doi:10.48550/arxiv.1805.08574 · Cited by 10
“mLSTM has been shown to improve upon RNNs and LSTMs on language modeling tasks (Krause et al., 2017; Radford et al., 2017).”
BackgroundMethodThe geometry of integration in text classification RNNsPreprint
Aitken, K., Ramasesh, V., Garg, A., Cao, Y., Sussillo, D., & Maheswaranathan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15114 · Cited by 11
“…auxiliary classifiers (e.g., part-of-speech) on RNN trajectories to probe the network’s representations; use of challenge sets to capture wider language phenomena than seen in natural corpora; and visualization of hidden unit activations as in Karpathy et al. (2015) and Radford et al. (2017).”
BackgroundMachine Learning Techniques for Software Quality Assurance: A SurveyPreprint
Omri, S., & Sinz, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14056 · Cited by 8
“Radford et al. [64] focus on understanding the properties of representations learned by byte-level recurrent language models for sentiment analysis.”
BackgroundAccelerating Antibody Design with Active LearningPreprint
Seo, S.-W., Kwak, M. W., Kang, E., Kim, C., Park, E. Y., Kang, T. H., & Kim, J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.09.12.507690 · Cited by 8
“P ( x ) can be decomposed to conditional probabilities of next word[31, 18] or character[32] at i -th position, x i , for given previous words, or characters, x i : Rather than modeling the probability of a entire sequence, approximating conditional probability of next token(word, character or…”
BackgroundShrestha, S. L. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377812.3382163 · Cited by 7
“Neural Network: We use Long Short Term Memory networks, variant of recurrent neural networks following success of many recent work [8, 11, 14].”
MethodChen, Z., Sun, X., Jiao, X., Lian, F., Kang, Z., Wang, D., & Xu, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30087 · Cited by 19
Topic Modeling to Extract Information from Nutraceutical Product Reviews
John, D. L., Kim, E., Kotian, K., Ong, K. Y., White, T., Gloukhova, L., Woodbridge, D. M.-K., & Ross, N. (2019) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc.2019.8651723 · Cited by 7
“on product reviews [20], large-scale data analysis using distributed computing [21], [22] and fake review detection [11].”
Background"How are you?" Estimation of anxiety, sleep quality, and mood using computational voice analysis
Kim, S., Kwon, N., O'Connell, H., Fisk, N., Ferguson, S., & Bartlett, M. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9175788 · Cited by 7
A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning
Islam, M. M., Aguilar, G., Ponnusamy, P., Mathialagan, C. S., Ma, C., & Guo, C. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.10 · Cited by 10
Khan, T., Mallick, D. D., Khan, M. S. I., Hasan, M. M., & Ashraf, F. B. (2022) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit57492.2022.10054834 · Cited by 7
“The authors investigated the properties of the byte-level recurrent language model [5].”
BackgroundMethodInfluentialZhang, Q., Guo, B., Wang, H., Liang, Y., Hao, S., & Yu, Z. (2019) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/smartworld-uic-atc-scalcom-iop-sci.2019.00176 · Cited by 7
“Radford et al. [47] demonstrated the direct influence of emotional units on the process of model generation.”
BackgroundThe Utility of Context When Extracting Entities From Legal Documents
Donnelly, J., & Roegiest, A. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412746 · Cited by 8
“Other works [6, 19] have focused on using linear classifiers on top of representations obtained from pre-trained language models.”
BackgroundJain, A., Aggarwal, I., & Singh, A. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2029 · Cited by 7
“Our solution uses multiple types of feature embeddings viz Skip-Thought vectors (Kiros et al., 2015), Unsupervised Sentiment Neuron (Radford et al., 2017), DeepMoji’s attention and last layer (Felbo et al., 2017) embedding along with Glove (Pennington et al., 2014), Emoji Em-bedding (Eisner et al.,…”
MethodExplaining Concept Drift of Deep Learning Models
Wang, X., Wang, Z., Shao, W., Jia, C., & Li, X. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37352-8_46 · Cited by 7
Object category learning and retrieval with weak supervisionPreprint
Hickson, S., Angelova, A., Essa, I. A., & Sukthankar, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.08985 · Cited by 7
“This has been explored by many representations including sequence to sequence learning and textual representations (Radford, Jozefowicz, and Sutskever (2017), Ramachandran, Liu, and Le (2016)).”
BackgroundComparison of Machine Learning Techniques for Multi-label Genre Classification
Pieters, M., & Wiering, M. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-76892-2_10 · Cited by 6
“Previous research has shown that first training a part of the model on an unsupervised task can reduce the training time and increase the accuracy on the supervised task [16].”
BackgroundLearning to Generate Multiple Style Transfer Outputs for an Input Sentence
Lin, K., Liu, M.-Y., Sun, M., & Kautz, J. (2020) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/2020.ngt-1.2 · Cited by 7
“Specifically, we apply the Byte-mLSTM (Radford et al., 2017) to classify the output sentence generated by a text style transfer model.”
MethodExplaining HCV prediction using LIME model
Gupta, S., & Sikka, G. (2021) · International Conference on Secure Cyber Computing and Communications (ICSCCC) · doi:10.1109/icsccc51823.2021.9478092 · Cited by 6
Recent Trends of Fake News Detection: A Review
Gupta, A., Anjum, A., Gupta, S., & Katarya, R. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-2354-7_43 · Cited by 6
Başgöl, H., Ayhan, İ., & Uğur, E. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101167 · Cited by 7
Jin, K., Zhang, X., & Zhang, J. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006577 · Cited by 6
“[31] develop a method to discover sentiment of generated reviews from Amazon product review dataset [18] and Ni et al.”
MethodDeflated reputation using multiplicative long short-term memory neural networks
Ma, Y., Zhang, Z., Li, D., & Tang, M. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2020.12.029 · Cited by 6
Automatically Exposing Problems with Neural Dialog Models
Yu, D., & Sagae, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.37 · Cited by 8
“Compared to sentiment neurons (Radford et al., 2017), learning trigger hidden states as a problem switch is a much harder task because our hidden states in a relative shallow model are not trained with a huge amount of clearly distinguished supervised data.”
BackgroundProduct Review Based on Facial Expression Detection
Bera, C., Adhav, P., Amati, S., & Singhaniya, N. G. (2022) · ITM Web of Conferences · doi:10.1051/itmconf/20224403061 · Cited by 6
Faster Training by Selecting Samples Using Embeddings
Gonzalez, S., Landgraf, J., & Miikkulainen, R. P. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851717 · Cited by 6
“Some networks can take a month to train, even on a multi-GPU system [1], so any improvement in this training time can result in a faster development cadence.”
BackgroundA journey in ESN and LSTM visualisations on a language taskPreprint
Variengien, A., & Hinaut, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01748 · Cited by 13
“Beyond being able to keep track of simple structure in a sentence such as shown by Karpathy, Johnson and Fei-Fei, individual cells of LSTM can also encode high level features such as positive or negative sentiment of a review, as shown in [15].”
BackgroundFrom Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning
Chen, W., Huang, Z., Xie, L., Lin, B., Li, H., Lu, L., Tian, X., Cai, D., Zhang, Y., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.01658 · Cited by 5
“Specifically, KL is calculated on a sub-set of Open Web Text (Radford et al., 2017) with 1000 texts truncated to a max length of 128 tokens.”
MethodTilmatine, M., Lüdtke, J., & Jacobs, A. M. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1431764 · Cited by 8
Generating Musical Sequences with a Given Emotion
Grekow, J. (2021) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc52423.2021.9658604 · Cited by 5
“It developed the method used for generating textual product reviews with a sentiment [4] by using a single-layer multiplicative long short-term memory (mLSTM) network.”
MethodInter and Intra Document Attention for Depression Risk Assessment
Maupomé, D., Queudot, M., & Meurs, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18305-9_27 · Cited by 5
“The flexibility of the transition function in mLSTM has shown to be capable of arriving at highly abstract features on its own and can achieve competitive results in sentiment analysis [21].”
BackgroundGonzález, J.-Á., Hurtado, L.-F., & Plà, F. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2031 · Cited by 5
“It would be also interesting to incorporate other emotional or sentiment features such as: Sentiment Unit (Rad-ford et al., 2017), DeepMoji (Felbo et al., 2017), Sentiment Specific WE (Tang et al., 2014); or polarity lexicons.”
BackgroundTwitter Sentiment Analysis in Under-Resourced Languages using Byte-Level Recurrent Neural Model
Ferdiana, R., Dicka, W. F., Dwi, D., Ayu, A. S. T., & Jatmiko, F. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0100815 · Cited by 5
“[14] proposed the learning of byte-level text representation.”
BackgroundComparative Analysis of Machine Learning Algorithms for Sentiment Classification in Amazon Reviews
Yu, B. (2024) · Highlights in Business Economics and Management · doi:10.54097/eqmavw44 · Cited by 5
“Furthermore, an exploration into the application of machine learning algorithms on the Stanford Sentiment Treebank dataset revealed that models can achieve high accuracy even with limited labeled data [8].”
BackgroundContinuous Learning in a Hierarchical Multiscale Neural Network
Wolf, T., Chaumond, J., & Delangue, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2001 · Cited by 6
“Recently, the task of language modeling has been shown to be an adequate proxy for learning unsupervised representations of high-quality in tasks like text classification (Howard and Ruder, 2018), sentiment detection (Radford et al., 2017) or word vector learning (Peters et al.”
BackgroundBickel, D. R. (2021) · Communication in Statistics- Theory and Methods · doi:10.1080/03610926.2020.1863988 · Cited by 5
“That problem has been addressed by applying deep neural networks to movie reviews (Radford et al., 2017).”
BackgroundClustering Multilingual Aspect Phrases for Sentiment Analysis
Pessutto, L. R. C., Vargas, D. S., & Moreira, V. P. (2018) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1109/wi.2018.00-91 · Cited by 5
Cell-aware Stacked LSTMs for Modeling SentencesPreprint
Choi, J., Kim, T., & Lee, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02279 · Cited by 7
Cooperative Learning of Disjoint Syntax and SemanticsPreprint
Havrylov, S., Kruszewski, G., & Joulin, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09393 · Cited by 14
Surprisal-Triggered Conditional Computation with Neural NetworksPreprint
Lugosch, L., Nowrouzezahrai, D., & Meyer, B. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01659 · Cited by 9
“…whether to apply zoneout to units in LSTMs. Autoregressive neural network models have not only been used to predict future observations or as generative models; they have also been used more recently with great success as unsupervised pre-trained feature extractors [49, 50, 51, 52, 53, 54, 55].”
MethodPredicting word vectors for microtext
Chaturvedi, I., Satapathy, R., Lynch, C., & Cambria, E. (2024) · Expert Systems · doi:10.1111/exsy.13589 · Cited by 5
“Recent advances in neural language modelling (LM) modelled language to be distributions over sequences of characters (Graves, 2013; Kim et al., 2015; Radford et al., 2017).”
BackgroundMethodTextmining at EmoInt-2017: A Deep Learning Approach to Sentiment Intensity Scoring of English Tweets
Meisheri, H., Saha, R., Sinha, P., & Dey, L. (2017) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/w17-5226 · Cited by 4
“In addition to this, we use the SentiNeuron model proposed by (Radford et al., 2017) to generate another feature.”
MethodDhiman, R., Kang, G. S., & Gupta, V. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11210-6 · Cited by 4
“The proposed work works in this direction and proposes an emotion recognition system from speech using deep neural networks which are also being used in multiple areas such as to perform sentiment analysis based on the reviews [38], recognition of speech [24], machine translation systems [49] and is even used for expl…”
MethodExploring Differences in the Impact of Users’ Traces on Arabic and English Facebook Search
Badache, I. (2019) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3350546.3352522 · Cited by 4
“In our work, sentiment analysis is only a part of English social IR process, we used the approach proposed in [38] to measure comment polarity and consider it as an additional social feature.”
MethodXu, C., Wang, H., Wu, S., & Lin, Z. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115182 · Cited by 5
Zarecki, J., & Markovitch, S. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/369 · Cited by 5
“For SST, CMR, SUBJ & KS we used the open-source implementation of Generating Reviews and Discovering Sentiment [23], which achieved state-of-the-art results for CMR and SUBJ.”
MethodHierarchical Self-Attention Hybrid Sparse Networks for Document Classification
Huang, W., Tao, Z., Huang, X., Xiong, L., & Yu, J. (2021) · Mathematical Problems in Engineering · doi:10.1155/2021/5594895 · Cited by 4
“Mikhail and Nikunj [15] proposed the à la carte embedding based on byte-level recurrent language models [16] achieve impressive efficiency on results.”
BackgroundSuffix Bidirectional Long Short-Term Memory.Preprint
Brahma, S. (2018) · arXiv (Cornell University) · Cited by 4
Low-Rank RNN Adaptation for Context-Aware Language ModelingPreprint
Jaech, A., & Mari Ostendorf, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.02603 · Cited by 4
“This finding is consistent with previous work that showed that individual dimensions of LSTM hidden states can be strong indicators of concepts like sentiment (Karpathy et al., 2015; Radford et al., 2017).”
ResultFineText: Text Classification via Attention-based Language Model Fine-tuningPreprint
Tao, Y., Gupta, S., Krishna, S., Zhou, X. H., Majumder, O., & Khare, V. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11959 · Cited by 4
New frontiers in AI for biodiversity research and conservation with multimodal language modelsPreprint
Miao, Z., Zhang, Y., Fabian, Z., Celis, A. H., Beery, S., Li, C., Liu, Z., Gupta, A., Nasir, M., Li, W., et al. (2024) · doi:10.32942/x22s6f · Cited by 4
An Empirical Evaluation of Word Embedding Models for Subjectivity Analysis Tasks
Nandi, R., Maiya, G., Kamath, P. R., & Shekhar, S. (2021) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect49130.2021.9392437 · Cited by 4
“Several models have been suggested for performing subjectivity analysis [20, 28, 5, 18], however the AdaSent model remains the state-of-the-art model in this domain.”
BackgroundMethodFox, I. (2020) · Deep Blue (University of Michigan) · Cited by 3
Property Neurons in Self-Supervised Speech Transformers
Lin, T.-Q., Lin, G.-T., Lee, H.-Y., & Tang, H. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832151 · Cited by 3
“Other than visualizing the filters, one can also identify the input that leads to high response for a particular neuron [15], or more generally, correlate the response of a neuron with properties of the input [16, 17, 18].”
BackgroundLow-Rank Approximations of Second-Order Document Representations
Lagus, J., Sinkkonen, J., & Arto Klami, A. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1059 · Cited by 4
“Even though there is extensive literature on using, e.g., character-level and other sub-word information (Lee et al., 2017; Radford et al., 2017; Mikolov et al., 2018) or non-Euclidean embedding spaces (Nickel and Kiela, 2017; Muzellec and Cuturi, 2018), the standard embeddings remain currently as…”
BackgroundCustomer Emotions Recognition Using Facial and Textual Review
Thakre, P., & Agarkar, P. (2020) · Cited by 3
Related Tasks Can Share! A Multi-task Framework for Affective Language
Deep, K. S., Akhtar, M. S., Ekbal, A., & Bhattacharyya, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24340-0_18 · Cited by 3
“– Unsupervised Sentiment Neuron: [23] developed an unsupervised system which learned an excellent representation of sentiment.”
BackgroundMethodKaghazgaran, P., Wang, J., Huang, R., & Caverlee, J. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401074 · Cited by 3
Unsupervised Learning of Explainable Parse Trees for Improved Generalisation
Sahay, A., Maheshwari, A., Kumar, R., Ramakrishnan, G., Hanawal, M. K., & Arya, K. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533448 · Cited by 3
Learning to Embed Sentences Using Attentive Recursive Trees
Shi, J., Hou, L., Li, J., Liu, Z., & Zhang, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016991 · Cited by 3
Emotion Detection using Social Media Data
Mehta, A. (2021) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2021.39027 · Cited by 3
“[4 ] [7] Social platforms have become increasingly important in people's daily lives as networks have grown.”
BackgroundRVAE-LAMOL: Residual Variational Autoencoder to Enhance Lifelong Language Learning
Wang, H., Fu, R., Zhang, X., & Zhou, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892670 · Cited by 3
“To evaluate the capability of RVAE-LAMOL on diverse sequence generation tasks, we pick the following three tasks from DecaNLP [7] following LAMOL: English Wizard of Oz (WOZ) [33] is a dataset of oriented dialogue of the restaurant reservation task; QA-SRL [34] is a dataset of semantic role labeling in SQuAD-style; SST…”
MethodCreating Semantic Representations
Nielsen, F. Å., & Hansen, L. K. (2020) · Springer eBooks · doi:10.1007/978-3-030-37250-7_2 · Cited by 3
“For instance, provided with large amount of text data, such as 82 million product reviews from Amazon, a UTF-8 encoded byte-level language model with a recurrent neural network may be trained for one month [38].”
BackgroundLearning Robust, Transferable Sentence Representations for Text ClassificationPreprint
Ahmad, W. U., Bai, X., Peng, N., & Chang, K.-W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00681 · Cited by 3
“Byte mLSTM (Radford et al., 2017) 86.9 91.4 94.6 88.5 - - - 75.0/82.8 2.”
InfluentialA Similarity Score Model for Aspect Category Detection
Madhoushi, Z., Hamdan, A. R., & Zainudin, S. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120738 · Cited by 2
Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets
Jiang, H., Wang, R., Wei, Z., Li, Y., & Wang, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.375 · Cited by 6
Unsupervised Natural Question Answering with a Small Model
Andrews, M., & Witteveen, S. (2019) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/d19-6606 · Cited by 2
“As mentioned in Sutskever (2019), an online video in which Radford et al. (2017) is discussed, ‘higher order’ capabilities seem to appear in language-related models only if the size of the model is sufficient to have captured many the basic features of the underlying language, since knowing the…”
BackgroundA system for the 2019 Sentiment, Emotion and Cognitive State Task of DARPA's LORELEI project
Martinez, V. R., Ramakrishna, A., Chiu, M.-C., Singla, K., & Narayanan, S. S. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925499 · Cited by 2
“To identify sentiment expressed towards an SF, we use the pretrained model described in [46], in which a multiplicative LSTM cell is trained at the character level on a corpus”
MethodLow-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
Fanthomme, A., & Monasson, R. (2021) · Neural Computation · doi:10.1162/neco_a_01366 · Cited by 2
“…studies have focused on the emergence of disentangled representations, which have been shown to be relevant in both Natural Language Processing (Radford et al., 2017) and Computer Vision (Denton and Birodkar, 2017; Lee et al., 2018), suggesting that the optimal type of representation might…”
BackgroundChen, J., Card, D., & Jurafsky, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.288 · Cited by 3
Contradiction in Reviews: is it Strong or Low?
Badache, I., Fournier, S., & Chifu, A.-G. (2018) · HAL · Cited by 2
Karasalo, M., Nilsson, M., Rosell, M., & Bolin, U. W. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1014 · Cited by 2
“In (Radford et al., 2017), the authors consider the problem of predicting the next character in a text given the preceding characters.”
BackgroundMoonGrad at SemEval-2019 Task 3: Ensemble BiRNNs for Contextual Emotion Detection in Dialogues
Bothe, C., & Wermter, S. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2044 · Cited by 2
“On the other hand, the EmoContext dataset is collected from the social media, and so the character language model used in our experiments is also trained on such a corpus (Radford et al., 2017).”
BackgroundMethodOpinion Mining: Is Feature Engineering Still Relevant?
Rahman, M. A., & Chakraborty, P. (2021) · International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD) · doi:10.1109/icict4sd50815.2021.9396874 · Cited by 2
“This incredible feat of performance (91.8% accuracy) was achieved by openAI’s Unsupervised Sentiment Neuron system on the Stanford Sentiment Treebank (SST) corpus [9].”
MethodInfluentialPearlman, E. (2021) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3452918.3467814 · Cited by 3
“GPT-2 outperforms state-of-the-art language modeling scores known as “zero-shot” settings from any other known language models currently in use (See Radford [4]).”
BackgroundA Dynamic Convolutional Neural Network Approach for Legal Text Classification
Hammami, E., Faïz, R., & Akermi, I. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-85977-0_6 · Cited by 2
Towards Language Agnostic Universal Representations
Aghajanyan, A., Song, X., & Tiwary, S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1395 · Cited by 2
“Even further, Radford et al. (2017) show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured…”
BackgroundWhat Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons
Xin, J., Lin, J., & Yu, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1591 · Cited by 3
“…low-level information stored in individual LSTM neurons, such as linguistic properties (Bau et al., 2019; Qian et al., 2016), syntax of source code (Karpathy et al., 2015), and sentiment (Radford et al., 2017), how each neuron contributes directly to the final classification layer remains unclear.”
BackgroundSemi-Supervised Lifelong Language Learning
Zhao, Y., Zheng, Y., Yu, B., Tian, Z., Lee, D., Sun, J., Li, Y., & Zhang, N. L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.290 · Cited by 2
Improving Context Aware Language ModelsPreprint
Jaech, A., & Mari Ostendorf, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06380 · Cited by 2
“Furthermore, we find that a single dimension in the hidden state of the unadapted model is often enough to distinguish between different languages even though the model was not given any supervision signal (Karpathy et al., 2015; Radford et al., 2017).”
BackgroundMeta-Learning a Dynamical Language ModelPreprint
Wolf, T., Chaumond, J., & Delangue, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10631 · Cited by 4
“…sensible loss function for learning high-quality unsupervised representations in tasks like text classification (Howard & Ruder, 2018), sentiment detection (Radford et al., 2017) or word vector learning (Peters et al., 2018) and there is thus a revived interest in developing better language models.”
BackgroundLarge‐Scale Social Multimedia Analysis
Bischke, B., Borth, D., & Dengel, A. (2019) · doi:10.1002/9781119376996.ch6 · Cited by 2
A Comparative Analysis of Knowledge-Intensive and Data-Intensive Semantic ParsersPreprint
Cao, J., Lin, Z. J., Sun, W., & Wan, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02298 · Cited by 2
“Related work tries to answer these questions by: (1) investigating specific components of the architectures (Karpathy, Johnson, and Fei-Fei 2015; Radford, Jozefowicz, and Sutskever 2017; Qian, Qiu, and Huang 2016; Bau et al. 2018), (2) testing models on specific tasks, including part-of-speech…”
BackgroundToward estimating personal well-being using voicePreprint
Kim, S., Kwon, N., & O'Connell, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10082 · Cited by 2
“2We built a dictionary with depression-related words and negative expressions, as well as common words observed from depression patients’ speech, which resulted in 486 terms [19] 3The sentiment likelihood score is generated by the binary classification model trained on Stanford Sentiment Treebank using Sentiment Neuro…”
MethodNeural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random UnitsPreprint
Meftah, S., Semmar, N., Tamaazousti, Y., Essafi, H., & Sadat, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04935 · Cited by 2
“…cells learned in language modelling and found multiple interpretable units that track long-distance dependencies, such as line lengths and quotes; Radford et al. (2017) visualised a unit which performs sentiment analysis in a language model based on Recurrent Neural Networks (RNNs); Bau et al.…”
BackgroundLinking in Style: Understanding Learned Features in Deep Learning Models
Wehrheim, M. H., Osuna-Vargas, P., & Kaschube, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_10 · Cited by 1
User Intent-Based Music Generation Model Combining Actor-Critic Approach With MusicVAE
Jang, S., & Lee, J. (2025) · IEEE Access · doi:10.1109/access.2025.3597741 · Cited by 1
Affect in Tweets Using Experts Model
Oota, S. R., Avvaru, A., Marreddy, M., & Mamidi, R. (2019) · Institutional Repositories DataBase (IRDB) · doi:10.48550/arxiv.1904.00762 · Cited by 1
“Unsupervised sentiment neuron model (Radford et al., 2017) provides an excellent learning representation of sentiment, despite being trained only on the text of Amazon reviews.”
BackgroundFake News Detection via Sentiment Neutralization
Huang, H., Peng, H.-T., & Pao, H.-K. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386174 · Cited by 3
“Kumari et al. [11] used BERT and the proposed method from Radford et al. [19] to extract emotion and sentiment labels, and integrate the two representations to enhance fake news detection performance.”
MethodAssessing Quality Estimation Models for Sentence-Level Prediction
Cuong, H. M., & Xu, J. (2018) · International Conference on Computational Linguistics · Cited by 1
Multiplicative Models for Recurrent Language Modeling
Maupomé, D., & Meurs, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24337-0_24 · Cited by 1
“Along this line of thought, previous work on mRNNs demonstrated their increased potential when compared to their regular variants [23,15,21].”
BackgroundKagawa, R., Baba, Y., & Tsurushima, H. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671418 · Cited by 1
Allerhand, L., Youngmann, B., Yom‐Tov, E., & Arkadir, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01138 · Cited by 1
“In essence, the authors of [14] used character-lever predictions to extract high-level features (e.g. sentiment) from documents.”
BackgroundMethodResultInfluentialUsing General Adversarial Networks for Marketing: A Case Study of AirbnbPreprint
Martinez, R. D., & Kamalu, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.11432 · Cited by 1
“al who underscore the importance of training word embeddings on a data-specific corpus for best results on generative tasks [9].”
BackgroundAspect Based Sentiment Analysis On Review Data
Xue, W. (2017) · FIU - Digital Commons (Florida International University) · doi:10.25148/etd.fidc004076 · Cited by 2
Metadata
- Authors
- Radford, A., Jozefowicz, R., & Sutskever, I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.01444
- OpenAlex
- W2606347107
- Cited by
- 330
- References
- 47
- Citations by year
- 2025: 3 · 2024: 11 · 2023: 16 · 2022: 16 · 2021: 52 · 2020: 58 · 2019: 84 · 2018: 63 · 2017: 24
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- sentiment analysis, unsupervised feature learning, disentangled representations, generative text, data efficiency, Stanford Sentiment Treebank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
