Cloud-Based Sentiment Analysis for Interactive Agents
Open accessAbstract
Emotions play an important role in human-agent interaction. To realise natural interaction it is essential for an agent to be able to analyse the sentiment in users' utterances. Modern agents use a distributed service model in which their functions can be located on any number of computers including cloud-based servers. Outsourcing the speech recognition and sentiment analysis to a cloud service enables even simple agents to adapt their behaviour to the emotional state of their users. In this study we test whether sentiment analysis tools can accurately gauge sentiment in human-chatbot interaction. To that effect, we compare the quality of sentiment analysis obtained from three major suppliers of cloud-based sentiment analysis services (Microsoft, Amazon and Google). In addition, we compare their results with the leading lexicon-based software, as well as with human ratings. The results show that although the sentiment analysis tools agree moderately with each other, they do not correlate well with human ratings. While the cloud-based services would be an extremely useful tool for human-agent interaction, their current quality does not justify their usage in human-agent conversations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
(2016) · Cited by 11,679 · see citing works
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771 · see citing works
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024 · see citing works
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Lexicon-based methods for sentiment analysis
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011) · Computational Linguistics · doi:10.1162/coli_a_00049 · Cited by 3,301 · see citing works
Sentiment analysis algorithms and applications: A survey
Medhat, W., Hassan, A., & Korashy, H. (2014) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2014.04.011 · Cited by 3,265 · see citing works
Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., & Qin, B. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1146 · Cited by 1,181 · see citing works
Deep Pyramid Convolutional Neural Networks for Text Categorization
Johnson, R., & Zhang, T. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1052 · Cited by 862 · see citing works
Russell, J. A. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.3.345 · Cited by 857 · see citing works
Sentiment analysis using product review data
Fang, X., & Zhan, J. Z. (2015) · Journal Of Big Data · doi:10.1186/s40537-015-0015-2 · Cited by 791 · see citing works
Schuller, B. W., Batliner, A. M., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 765 · see citing works
Twitter Sentiment Analysis with Deep Convolutional Neural Networks
Severyn, A., & Moschitti, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767830 · Cited by 673 · see citing works
How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews
Guzmán, E., & Maalej, W. (2014) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2014.6912257 · Cited by 611 · see citing works
SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
Ribeiro, F. N., Araújo, M., Gonçalves, P., André Gonçalves, M., & Benevenuto, F. (2016) · EPJ Data Science · doi:10.1140/epjds/s13688-016-0085-1 · Cited by 468 · see citing works
Weller, K., Bruns, A., Burgess, J., Mahrt, M., & Puschmann, C. (2014) · doi:10.3726/978-1-4539-1170-9 · Cited by 333 · see citing works
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317 · see citing works
Self-citationPredicting Fluctuations in Cryptocurrency Transactions Based on User Comments and Replies
Kim, Y., Kim, J. G., Kim, W., Im, J., Kim, T. H., Kang, S.-J., & Kim, C. (2016) · PLoS ONE · doi:10.1371/journal.pone.0161197 · Cited by 287 · see citing works
Gilpin, A. R. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053001007 · Cited by 223 · see citing works
Sharing feelings online: studying emotional well-being via automated text analysis of Facebook posts
Settanni, M., & Marengo, D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01045 · Cited by 211 · see citing works
Validating Machine Learning Algorithms for Twitter Data Against Established Measures of Suicidality
Braithwaite, S., Giraud-Carrier, C. G., West, J. H., Barnes, M. D., & Hanson, C. L. (2016) · JMIR Mental Health · doi:10.2196/mental.4822 · Cited by 193 · see citing works
What in Consumer Reviews Affects the Sales of Mobile Apps: A Multifacet Sentiment Analysis Approach
Liang, T., Li, X., Yang, C.-T., & Wang, M. (2015) · International Journal of Electronic Commerce · doi:10.1080/10864415.2016.1087823 · Cited by 163 · see citing works
Survey of Non-facial/Non-verbal Affective Expressions for Appearance-Constrained Robots
Bethel, C. L., & Murphy, R. R. (2007) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.905845 · Cited by 154 · see citing works
On the Personality Traits of StackOverflow Users
Bazelli, B., Hindle, A., & Stroulia, E. (2013) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2013.72 · Cited by 109 · see citing works
#MeToo Alexa: How Conversational Systems Respond to Sexual Harassment
Curry, A. C., & Rieser, V. (2018) · doi:10.18653/v1/w18-0802 · Cited by 87 · see citing works
Emotional conflict in interpersonal interactions
Ruz, M., & Tudela, P. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.08.039 · Cited by 44 · see citing works
Twitter and society [Digital Formations, Volume 89]
Weller, K., Bruns, A., Burgess, J., Mahrt, M., & Puschmann, C. (2014) · QUT ePrints (Queensland University of Technology) · Cited by 18 · see citing works
Flagged first, then by citations
1 of the 29 references couldn’t be matched to a known paper.
Research works printed but not linked: 10
- Christoph Bartneck and Michael J. Lyons. 2009. Facial Expression Analysis, Mod- eling and Synthesis: Overcoming the Limitations of Artificial Intelligence with the Art of the Soluble. In Handbook of Research on Synthetic Emotions and Sociable Robotics:New Applications in Affective Computing and Artificial Intelligence, Jordi Vallverdu and David Casacuberta (Eds.). IGI Global, Chapter 3, 33-53.
- Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems. 4349-4357.
- Henriette Cramer, Jorrit Goddijn, Bob Wielinga, and Vanessa Evers. 2010. Effects of (in) accurate empathy and situational valence on attitudes towards robots. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on. IEEE, 141-142.
- Golnoosh Farnadi, Susana Zoghbi, Marie-Francine Moens, and Martine De Cock. 2013. Recognising personality traits using Facebook status updates. In Proceedings of the workshop on computational personality recognition (WCPR13) at the 7th international AAAI conference on weblogs and social media. https://www.aaai. org/ocs/index.php/ICWSM/ICWSM13/paper/view/6245/6309
- Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th international conference on machine learning. 513-520. http://www. icml-2011.org/papers/342_icmlpaper.pdf
- Eva Hudlicka. 2008. What are we modeling when we model emotion?. In AAAI spring symposium: emotion, personality, and social behavior, Vol. 8. AAAI. http: //www.aaai.org/Papers/Symposia/Spring/2008/SS-08-04/SS08-04-010.pdf
- James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015. Technical Report. The University of Texas at Austin. http://hdl.handle.net/2152/31333
- Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444 (2017). · arXiv
- Sebastian Ruder and Barbara Plank. 2018. Strong Baselines for Neural Semi- supervised Learning under Domain Shift. arXiv preprint arXiv:1804.09530 (2018). · arXiv
- Lei Zhang, Riddhiman Ghosh, Mohamed Dekhil, Meichun Hsu, and Bing Liu. 2011. Combining lexicon-based and learning-based methods for twitter sentiment analysis. Technical Report 89. HP Laboratories. https://uic.pure.elsevier.com/en/publications/ combining-lexicon-based-and-learning-based-methods-for-twitter-se
Cited by
Sorted by citations
Chatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090 · see citing works
Conversational Affective Social Robots for Ageing and Dementia Support
Lima, M. R., Wairagkar, M., Gupta, M., Rodriguez y Baena, F., Barnaghi, P., Sharp, D. J., & Vaidyanathan, R. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3115228 · Cited by 75 · see citing works
Commercial Use of Emotion Artificial Intelligence (AI): Implications for Psychiatry
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01330-7 · Cited by 61 · see citing works
Struggling to Keep Tabs on Capstone Projects: A Chatbot to Tackle Student Procrastination
Pereira, J., & Díaz, Ó. (2021) · ACM Transactions on Computing Education · doi:10.1145/3469127 · Cited by 31 · see citing works
Keijsers, M., Bartneck, C., & Eyssel, F. (2021) · Interaction Studies Social Behaviour and Communication in Biological and Artificial Systems · doi:10.1075/is.20002.kei · Cited by 29 · see citing works
Self-citationSentiment Analysis using Deep Learning in Cloud
Raza, M. R., Hussain, W., Tanyıldızı, E., & Varol, A. (2021) · International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs52919.2021.9486312 · Cited by 26 · see citing works
Singh, J., Sanga, P., Khanna, N. N., Laird, J. R., Singh, I. M., Kalra, M. K., Mantella, L. E., Johri, A. M., Faa, G., Tiwari, E., et al. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02541-w · Cited by 4 · see citing works
Laban, G. (2024) · doi:10.31234/osf.io/fhdek · Cited by 4 · see citing works
Classification and Prospects for HAI Research Based on a Tentative Framework
Sakamoto, T., & Takeuchi, Y. (2020) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3406499.3415085 · Cited by 2 · see citing works
- Authors
- Keijsers, M., Bartneck, C., & Kazmi, H. S.
- Affiliations
- University of Canterbury (NZ)
- KU Leuven (BE)
- Venue
- Proceedings of the 7th International Conference on Human-Agent Interaction
- Publisher
- ACM
- Published
- 2019-09-25
- Pages
- 43-50
- OpenAlex
- W2978595383
- Cited by
- 9
- References
- 29
- Citations by year
- 2025: 1 · 2024: 1 · 2022: 1 · 2021: 4 · 2020: 2
- FWCI
- 0.67
- Type
- conference paper
- Language
- en
- Open access
- green
- lirias.kuleuven.be
- Locations
- International Conference on Human-Agent Interaction (HAI) (publishedVersion)
- Lirias (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- sentiment analysis, human-chatbot interaction, human-agent interaction, emotion recognition, lexicon-based sentiment analysis, human ratings, service quality evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
