CoChat
Sign Up
Cited by 14Hate Speech and Cyberbullying DetectionWorkplace Violence and BullyingMisinformation and Its Impacts

Users’ Behavioral and Emotional Response to Toxicity in Twitter Conversations

Open access
Aleksandric, Ana, Saha Roy, Sayak, Pankaj, Hanani, Wilson, Gabriela Mustata, Nilizadeh, Shirin·2024·Proceedings of the International AAAI Conference on Web and Social Media
DOI 10.1609/icwsm.v18i1.31295
Open this paper in CoChat — free accountOpen PDF

Abstract

Prior works have shown connections between online toxicity attacks, such as harassment, cyberbullying, and hate speech, and the subsequent increase in offline violence, as well as negative psychological effects on victims. These correlations are primarily identified through user studies conducted via virtual environments, simulations, and questionnaires. However, no work has investigated how, in practice and authentically, people react to online toxicity both emotionally, showing anger, anxiety, and sadness, and behaviorally in terms of engaging with and responding to toxicity instigators, considering conversations as a whole and the relation between emotions and behaviors. This data-driven study investigates the effect of toxicity on Twitter users' behaviors and emotions considering confounding factors, such as account identifiability, activity, and conversation's structure and topic. We collected about 80K Twitter conversations and identified those with and without toxic replies. Performing statistical tests along with propensity score matching, we investigated the causal association of receiving toxicity and users' responses. We found that authors of conversations with toxic replies are more likely to engage in conversations, reply in a toxic way, and unfollow toxicity instigators. In terms of users' emotional responses, we found that sadness and anger after the first toxic reply are more likely to increase as the amount of toxicity increases. These findings not only emphasize the negative emotional and behavioral effects of online toxicity on social media users but also, as demonstrated in this paper, can be utilized to build prediction models for users' reactions, which could then aid the implementation of proactive detection and intervention measures helping users in such situations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/icwsm.v18i1.31295

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The toxicity threshold is set at 0.61, identified by Kumar et al. (2021) and Aleksandric et al. (2024).

    “We set the threshold at 0.61 which is identified by Kumar et al. (2021) by employing a survey on 17,280 participants and Aleksandric et al. (2024) who found that Perspective API gets the highest accuracy on this threshold on their independent test set.”

    From Cross-Partisan Interactions on Twitter (2025) · cites this paper as Aleksandric et al. (2024)

    In “GPT Labels” (page not provided), Aleksandric et al. report that Perspective API achieved its highest accuracy at a threshold of 0.6, not 0.61, and say they used 0.6 to classify toxic replies. They compared the API against a manually labeled sample; the passage does not describe an independent test set. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 54

  1. Aleksandric, A.; Singhal, M.; Groggel, A.; and Nilizadeh, S. 2022. Understanding the Bystander Effect on Toxic Twitter Conversations. arXiv preprint arXiv:2211.10764. · arXiv
  2. Alhujailli, A.; Karwowski, W.; Wan, T. T.; and Hancock, P. 2020. Affective and stress consequences of cyberbullying. Symmetry, 12(9): 1536.
  3. Androcec, D. 2020. Machine learning methods for toxic comment classification: a systematic review. Acta Univer- sitatis Sapientiae, Informatica, 12(2): 205-216.
  4. Antypas, D.; Ushio, A.; Camacho-Collados, J.; Silva, V.; Neves, L.; and Barbieri, F. 2022. Twitter Topic Classifica- tion. In Proceedings of the 29th International Conference on Computational Linguistics, 3386-3400. Gyeongju, Re- public of Korea: International Committee on Computational Linguistics.
  5. Arató, N.; Zsidó, A.; Rivnyák, A.; Péley, B.; and Lábadi, B. 2022. Risk and protective factors in cyberbullying: the role of family, social support and emotion regulation. Interna- tional journal of bullying prevention, 4(2): 160-173.
  6. Armstrong, R. A. 2014. When to use the Bonferroni correc- tion. Ophthalmic and Physiological Optics, 34(5): 502-508.
  7. Beres, N. A.; Frommel, J.; Reid, E.; Mandryk, R. L.; and Klarkowski, M. 2021. Don't you know that you're toxic: Normalization of toxicity in online gaming. In Proceedings of the 2021 CHI Conference on Human Factors in Comput- ing Systems, 1-15.
  8. Bor, A.; and Petersen, M. B. 2022. The psychology of online political hostility: A comprehensive, cross-national test of the mismatch hypothesis. American political science review, 116(1): 1-18.
  9. Campbell, M.; Spears, B.; Slee, P.; Butler, D.; and Kift, S. 2012. Victims' perceptions of traditional and cyberbullying, and the psychosocial correlates of their victimisation. Emo- tional and Behavioural Difficulties, 17(3-4): 389-401.
  10. Chong, Y. Y.; and Kwak, H. 2022. Understanding toxicity triggers on Reddit in the context of Singapore. In Proceed- ings of the International AAAI Conference on Web and So- cial Media, volume 16, 1383-1387.
  11. Correa, D.; Silva, L. A.; Mondal, M.; Benevenuto, F.; and Gummadi, K. P. 2015. The many shades of anonymity: Characterizing anonymous social media content. In Ninth International AAAI Conference on Web and Social Media.
  12. Duffy, B. E.; and Hund, E. 2019. Gendered Visibility on So- cial Media: Navigating Instagram's Authenticity Bind. In- ternational Journal of Communication (19328036), 13.
  13. ElSherief, M.; Nilizadeh, S.; Nguyen, D.; Vigna, G.; and Belding, E. 2018b. Peer to peer hate: Hate speech instigators and their targets. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12.
  14. Eris ¸ti, B.; and Akbulut, Y. 2019. Reactions to cyberbully- ing among high school and university students. The Social Science Journal, 56(1): 10-20.
  15. Fredericks, B.; and Bradfield, A. 2021. 'Waiting with Bated Breath': Navigating the Monstrous World of Online Racism. M/C Journal, 24(5).
  16. Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.; Wallach, H.; Iii, H. D.; and Crawford, K. 2021. Datasheets for datasets. Communications of the ACM, 64(12): 86-92.
  17. Habib, H.; and Nithyanand, R. 2022. Exploring the Mag- nitude and Effects of Media Influence on Reddit Modera- tion. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, 275-286.
  18. Hannan, J. 2018. Trolling ourselves to death? Social media and post-truth politics. European Journal of Communica- tion, 33(2): 214-226.
  19. Horta Ribeiro, M.; Jhaver, S.; Zannettou, S.; Blackburn, J.; Stringhini, G.; De Cristofaro, E.; and West, R. 2021. Do plat- form migrations compromise content moderation? evidence from r/the donald and r/incels. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2): 1-24.
  20. Huang, F.; Kwak, H.; and An, J. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in ex- plaining implicit hate speech. In Companion proceedings of the ACM web conference 2023, 294-297.
  21. Kahn, J. H.; Tobin, R. M.; Massey, A. E.; and Anderson, J. A. 2007. Measuring emotional expression with the Lin- guistic Inquiry and Word Count. The American journal of psychology, 120(2): 263-286.
  22. Kowalski, R. M.; Giumetti, G. W.; Schroeder, A. N.; and Lattanner, M. R. 2014. Bullying in the digital age: A critical review and meta-analysis of cyberbullying research among youth. Psychological bulletin, 140(4): 1073.
  23. Kumar, D.; Hancock, J.; Thomas, K.; and Durumeric, Z. 2022. Understanding Longitudinal Behaviors of Toxic Ac- counts on Reddit. arXiv preprint arXiv:2209.02533. · arXiv
  24. Kumarswamy, N.; Singhal, M.; and Nilizadeh, S. 2023. Im- pact of Stricter Content Moderation on Parler's Users' Dis- course. arXiv preprint arXiv:2310.08844. · arXiv
  25. Kvålseth, T. O. 1989. Note on Cohen's kappa. Psychological reports, 65(1): 223-226.
  26. Kwan, I.; Dickson, K.; Richardson, M.; MacDowall, W.; Burchett, H.; Stansfield, C.; Brunton, G.; Sutcliffe, K.; and Thomas, J. 2020. Cyberbullying and children and young people's mental health: a systematic map of systematic re- views. Cyberpsychology, Behavior, and Social Networking, 23(2): 72-82.
  27. Lees, A.; Tran, V. Q.; Tay, Y.; Sorensen, J.; Gupta, J.; Met- zler, D.; and Vasserman, L. 2022. A new generation of perspective api: Efficient multilingual character-level trans- formers. In Proceedings of the 28th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, 3197- 3207.
  28. Maarouf, A.; Pröllochs, N.; and Feuerriegel, S. 2022. The Virality of Hate Speech on Social Media. arXiv preprint arXiv:2210.13770. · arXiv
  29. Mameli, C.; Menabò, L.; Brighi, A.; Menin, D.; Culbert, C.; Hamilton, J.; Scheithauer, H.; Smith, P. K.; Völlink, T.; Willems, R. A.; et al. 2022. Stay safe and strong: characteris- tics, roles and emotions of student-produced comics related to cyberbullying. International journal of environmental re- search and public health, 19(14): 8776.
  30. Mathew, B.; Illendula, A.; Saha, P.; Sarkar, S.; Goyal, P.; and Mukherjee, A. 2020. Hate begets hate: A temporal study of hate speech. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2): 1-24.
  31. Olteanu, A.; Castillo, C.; Boy, J.; and Varshney, K. 2018. The effect of extremist violence on hateful speech online. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12.
  32. Olteanu, A.; Varol, O.; and Kiciman, E. 2017. Distilling the Outcomes of Personal Experiences: A Propensity-Scored Analysis of Social Media. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW '17, 370-386. New York, NY, USA: Association for Computing Machinery. ISBN 9781450343350.
  33. Ortega, R.; Elipe, P.; Mora-Merchán, J. A.; Genta, M. L.; Brighi, A.; Guarini, A.; Smith, P. K.; Thompson, F.; and Tip- pett, N. 2012. The emotional impact of bullying and cyber- bullying on victims: A European cross-national study. Ag- gressive behavior, 38(5): 342-356.
  34. Palomino, M. A.; and Varma, A. P. 2020. Any Publicity is Good Publicity: Positive, Negative and Neutral Tweets Can All Become Trends. In 2020 39th International Con- ference of the Chilean Computer Science Society (SCCC), 1-8. IEEE.
  35. Parent, M. C.; Gobble, T. D.; and Rochlen, A. 2019. Social media behavior, toxic masculinity, and depression. Psychol- ogy of Men & Masculinities, 20(3): 277.
  36. Raskauskas, J.; and Stoltz, A. D. 2007. Involvement in tra- ditional and electronic bullying among adolescents. Devel- opmental psychology, 43(3): 564.
  37. Ribeiro, M.; Calais, P.; Santos, Y.; Almeida, V.; and Meira Jr, W. 2018. Characterizing and detecting hateful users on twit- ter. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12.
  38. Rosenbaum, P. R.; and Rubin, D. B. 1984. Reducing Bias in Observational Studies Using Subclassification on the Propensity Score. Journal of the American Statistical Asso- ciation, 79(387): 516-524.
  39. Salehabadi, N.; Groggel, A.; Singhal, M.; Roy, S. S.; and Nilizadeh, S. 2022. User Engagement and the Toxicity of Tweets. arXiv preprint arXiv:2211.03856. · arXiv
  40. Saveski, M.; Roy, B.; and Roy, D. 2021. The Structure of Toxic Conversations on Twitter. In Proceedings of the Web Conference 2021, 1086-1097.
  41. Schlesinger, A.; Chandrasekharan, E.; Masden, C. A.; Bruckman, A. S.; Edwards, W. K.; and Grinter, R. E. 2017. Situated anonymity: Impacts of anonymity, ephemerality, and hyper-locality on social media. In Proceedings of the 2017 CHI conference on human factors in computing sys- tems, 6912-6924.
  42. Silva, L.; Mondal, M.; Correa, D.; Benevenuto, F.; and We- ber, I. 2016. Analyzing the targets of hate in online social media. In Tenth International AAAI Conference on Web and Social Media.
  43. Tahmasbi, F.; Schild, L.; Ling, C.; Blackburn, J.; Stringh- ini, G.; Zhang, Y.; and Zannettou, S. 2021. "Go eat a bat, Chang!": On the Emergence of Sinophobic Behavior on Web Communities in the Face of COVID-19. In Proceed- ings of the web conference 2021, 1122-1133.
  44. Varjas, K.; Talley, J.; Meyers, J.; Parris, L.; and Cutts, H. 2010. High school students' perceptions of motivations for cyberbullying: An exploratory study. Western Journal of Emergency Medicine, 11(3): 269.
  45. Veletsianos, G.; Houlden, S.; Hodson, J.; and Gosse, C. 2018. Women scholars' experiences with online harassment and abuse: Self-protection, resistance, acceptance, and self- blame. New Media & Society, 20(12): 4689-4708.
  46. Wong, R. Y. M.; Cheung, C. M.; Xiao, B.; and Thatcher, J. B. 2021. Standing up or standing by: Understanding by- standers' proactive reporting responses to social media ha- rassment. Information Systems Research, 32(2): 561-581.
  47. Wright, V. H.; Burnham, J. J.; Inman, C. T.; and Ogorchock, H. N. 2009. Cyberbullying: Using Virtual Scenarios to Edu- cate and Raise Awareness. Journal of Computing in Teacher Education, 26(1): 35-42.
  48. Yang, X.; Ye, H. J.; and Wang, X. 2021. Social media use and work efficiency: Insights from the theory of communica- tion visibility. Information & Management, 58(4): 103462.
  49. Yao, X. I.; Wang, X.; Speicher, P. J.; Hwang, E. S.; Cheng, P.; Harpole, D. H.; Berry, M. F.; Schrag, D.; and Pang, H. H. 2017. Reporting and guidelines in propensity score analy- sis: a systematic review of cancer and cancer surgical stud- ies. JNCI: Journal of the National Cancer Institute, 109(8): djw323.
  50. Yousefi, N.; Noor, N. B.; Spann, B.; and Agarwal, N. 2023a. Examining Toxicity's Impact on Reddit Conversations. In International Conference on Complex Networks and Their Applications, 401-411. Springer.
  51. Zannettou, S.; ElSherief, M.; Belding, E.; Nilizadeh, S.; and Stringhini, G. 2020a. Measuring and characterizing hate speech on news websites. In Proceedings of the 12th ACM Conference on Web Science, 125-134.
  52. Zannettou, S.; Finkelstein, J.; Bradlyn, B.; and Blackburn, J. 2020b. A quantitative approach to understanding online an- tisemitism. In Proceedings of the International AAAI Con- ference on Web and Social Media, volume 14, 786-797.
  53. Zhang, K.; and Kizilcec, R. F. 2014. Anonymity in so- cial media: Effects of content controversiality and social endorsement on sharing behavior. In Eighth International AAAI Conference on Weblogs and Social Media.
  54. Zhang, Z.; Kim, H. J.; Lonjon, G.; Zhu, Y.; et al. 2019. Bal- ance diagnostics after propensity score matching. Annals of translational medicine, 7(1).

Cited by

Sorted by citations

Authors
Aleksandric, A., Saha Roy, S., Pankaj, H., Wilson, G. M., & Nilizadeh, S.
Affiliations
The University of Texas at Arlington (US)
ORCID
Venue
Proceedings of the International AAAI Conference on Web and Social Media
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2024-05-28
Volume
18
Pages
29-42
ISSN
  • 2162-3449 (ISSN-L)
  • 2334-0770
DOI
10.1609/icwsm.v18i1.31295
OpenAlex
W4399205929
Cited by
5
References
59
Citations by year
2026: 1 · 2025: 3 · 2024: 1
FWCI
1.44
Type
conference paper
Language
en
Open access
Locations
Proceedings of the International AAAI Conference on Web and Social Media (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Sociology and Political Science › Workplace Violence and Bullying
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
online toxicity, Twitter conversations, cyberbullying, harassment, hate speech, user emotional response, sadness, propensity score matching

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review