Examining Human-AI Collaboration for Co-Writing Constructive Comments Online
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
This paper examines if large language models (LLMs) can help people write constructive comments on divisive social issues due to the difficulty of expressing constructive disagreement online. Through controlled experiments with 600 participants from India and the US, who reviewed and wrote constructive comments on threads related to Islamophobia and homophobia, we observed potential misalignment between how LLMs and humans perceive constructiveness in online comments. While the LLM was more likely to prioritize politeness and balance among contrasting viewpoints when evaluating constructiveness, participants emphasized logic and facts more than the LLM did. Despite these differences, participants rated both LLM-generated and human-AI co-written comments as significantly more constructive than those written independently by humans. Our analysis also revealed that LLM-generated comments integrated significantly more linguistic features of constructiveness compared to human-written comments. When participants used LLMs to refine their comments, the resulting comments were more constructive, more positive, less toxic, and retained the original intent. However, LLMs often distorted people's original views--especially when their stances were on a spectrum instead of being outright polarizing. Based on these findings, we discuss ethical and design considerations in using LLMs to facilitate constructive discourse online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 1×
“To analyze sentiment, we used VADER sentiment analysis tool as it is attuned to sentiments expressed in social media [40].”
Culture, dialectics, and reasoning about contradiction.
Peng, K., & Nisbett, R. E. (1999) · American Psychologist · doi:10.1037//0003-066x.54.9.741 · Cited by 1,755 · cited 8×
“Prior research shows that people from individualistic cultures (e.g., the US) prefer logical reasoning during argumentation [71] while those from collectivist cultures (e.g., India) prefer dialectical arguments during conflict [76].”
InfluentialProceedings of the Web Conference 2021
(2021) · doi:10.1145/3442381 · Cited by 1,488
Proceedings of the ACM on Human-Computer Interaction
Karahalios, K. G., Monroy‐Hernández, A., Lampinen, A., & Fitzpatrick, G. (2018) · Cited by 1,074
Facework competence in intercultural conflict: an updated face-negotiation theory
Ting-toomey, S., & Kurogi, A. (1998) · International Journal of Intercultural Relations · doi:10.1016/s0147-1767(98)00004-2 · Cited by 864 · cited 4×
“Linguistic Features.”
InfluentialProceedings of 48th Hawaii International Conference on System Sciences (HICSS)
Pöyry, E., Parvinen, P. M. T., & McFarland, R. (2015) · Cited by 643
Building Successful Online Communities: Evidence-Based Social Design
Kraut, R. E., Resnick, P., Kiesler, S. B., Ren, Y., Chen, Y., Burke, M., Kittur, N., Riedl, J., & Konstan, J. A. (2012) · Cited by 636
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
(2025) · doi:10.1145/3706598 · Cited by 534
Universals of linguistic politeness
Hill, B., Ide, S., Ikuta, S., Kawasaki, A., & Ogino, T. (1986) · Journal of Pragmatics · doi:10.1016/0378-2166(86)90006-8 · Cited by 437
Research works printed but not linked: 44
- George Berry and Sean J. Taylor. 2017. Discussion Quality Diffuses in the Digital Public Square. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1371-1380.
- Emilie Bossens, Elias Storms, and David Geerts. 2021. Improving the Debate: Interface Elements that Enhance Civility and Relevance in Online News Comments. In IFIP Conference on Human-Computer Interaction. Springer, Switzerland, 433-450.
- Jonathan P. Chang, Charlotte Schluger, and Cristian Danescu-Niculescu-Mizil. 2022. Thread With Caution: Proactively Helping Users Assess and Deescalate Tension in Their Online Discussions. Proc. ACM Hum.-Comput. Interact., Vol. 6, CSCW2, Article 545 (nov 2022), 37 pages.
- Incheol Choi, Minkyung Koo, and Jong An Choi. 2007. Individual differences in analytic versus holistic thinking. Personality and social psychology bulletin, Vol. 33, 5 (2007), 691-705.
- Christine Linda Cook, Juliette Schaafsma, Marjolijn L Antheunis, Suleman Shahid, Jih-Hsuan Tammy Lin, and Hanne W Nijtmans. 2021. Trolls without borders: A cross-cultural examination of victim reactions to verbal and silent aggression online. Frontiers in psychology, Vol. 12 (2021), 549955.
- Stephanie Cutler Yu Xia and Kacey Beddoes. 2022. A Growth Mindset for Peer Review: Guidelines for writing constructive peer reviews.
- Debarati Das, Karin De Langis, Anna Martin, Jaehyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Hayati, Risako Owan, Bin Hu, Ritik Parkar, et al., 2024. Under the surface: Tracking the artifactuality of llm-generated data.
- Christine De Kock and Andreas Vlachos. 2021. I Beg to Differ: A study of constructive disagreement in online conversations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 2017-2027.
- Nicholas Diakopoulos. 2015. Picking the NYT picks: Editorial criteria and automation in the curation of online news comments. ISOJ Journal, Vol. 6, 1 (2015), 147-166.
- Xiaohan Ding, Kaike Ping, Uma Sushmitha Gunturi, Buse Carik, Sophia Stil, Lance T Wilhelm, Taufiq Daryanto, James Hawdon, Sang Won Lee, and Eugenia H Rho. 2024. CounterQuill: Investigating the Potential of Human-AI Collaboration in Online Counterspeech Writing.
- Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-llm collaboration.
- Catherine Grevet, Loren G. Terveen, and Eric Gilbert. 2014. Managing political differences in social media. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, New York, USA, 1400-1408.
- Selin Gurgun, Emily Arden-Close, John McAlaney, Keith Phalp, and Raian Ali. 2023. Can We Re-design Social Media to Persuade People to Challenge Misinformation? An Exploratory Study. In International Conference on Persuasive Technology. Springer, Germany, 123-141.
- David Hitchcock and Bart Verheij. 2006. Arguing on the Toulmin model. Vol. 10. Springer, New York, USA.
- 10.1037/0022-3514.59.4.719 · doi:10.1037/0022-3514.59.4.719 · Paper page
- Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2017. SpaCy: Industrial-strength natural language processing in Python.
- Elise Karinshak, Sunny Xun Liu, Joon Sung Park, and Jeffrey T. Hancock. 2023. Working With AI to Persuade: Examining a Large Language Model's Ability to Generate Pro-Vaccination Messages. Proc. ACM Hum.-Comput. Interact., Vol. 7, CSCW1, Article 116 (2023), 29 pages.
- Whose Knowledge? Oxford Internet Institute and The Centre for Internet & Society (India). 2022. State of the Internet's Languages.
- Varada Kolhatkar and Maite Taboada. 2017a. Constructive Language in News Comments. In Proceedings of the First Workshop on Abusive Language Online, Zeerak Waseem, Wendy Hui Kyong Chung, Dirk Hovy, and Joel Tetreault (Eds.). ACL, Vancouver, Canada, 11-17.
- Varada Kolhatkar and Maite Taboada. 2017b. Using New York Times Picks to Identify Constructive Comments. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, Octavian Popescu and Carlo Strapparava (Eds.). ACL, Copenhagen, Denmark, 100-105.
- 10.1145/3491102.3502030 · DOI · Paper page
- Yingqin Liu. 2009. The impact of cultural factors on Chinese and American college students' rhetorical choices in argumentative discourse: A contrastive study. Intercultural Communication Studies, Vol. 18, 1 (2009), 128.
- Teale W. Masrani, Jack Jamieson, Naomi Yamashita, and Helen Ai He. 2023. Slowing it Down: Towards Facilitating Interpersonal Mindfulness in Online Polarizing Conversations Over Social Media. Proc. ACM Hum.-Comput. Interact., Vol. 7, CSCW1, Article 90 (2023), 27 pages.
- Jimin Mun, Emily Allaway, Akhila Yerukola, Laura Vianna, Sarah-Jane Leslie, and Maarten Sap. 2023. Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9759-9777.
- Jimin Mun, Cathy Buerger, Jenny T Liang, Joshua Garland, and Maarten Sap. 2024. Counterspeakers' Perspectives: Unveiling Barriers and AI Needs in the Fight against Online Hate. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24). ACM, New York, USA, Article 742, 22 pages.
- Courtney Napoles, Joel Tetreault, Aasish Pappu, Enrica Rosato, and Brian Provenzale. 2017. Finding Good Conversations Online: The Yahoo News Annotated Comments Corpus. In Proceedings of the 11th Linguistic Annotation Workshop, Nathan Schneider and Nianwen Xue (Eds.). Association for Computational Linguistics, Valencia, Spain, 11 pages.
- Vlad Niculae and Cristian Danescu-Niculescu-Mizil. 2016. Conversational Markers of Constructive Discussions. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Kevin Knight, Ani Nenkova, and Owen Rambow (Eds.). Association for Computational Linguistics, San Diego, California, 568-578.
- Richard E Nisbett, Kaiping Peng, Incheol Choi, and Ara Norenzayan. 2001. Culture and systems of thought: holistic versus analytic cognition. Psychological review, Vol. 108, 2 (2001), 291.
- Ara Norenzayan, Edward E Smith, Beom Jun Kim, and Richard E Nisbett. 2002. Cultural preferences for formal versus intuitive reasoning. Cognitive science, Vol. 26, 5 (2002), 653-684.
- Kaiping Peng. 1997. Naive dialecticism and its effects on reasoning and judgment about contradiction. Ph.D. Dissertation. University of Michigan.
- 10.1037/0003-066X.54.9.741 · doi:10.1037/0003-066x.54.9.741 · Paper page
- Ashwin Rajadesingan, Carolyn Duran, Paul Resnick, and Ceren Budak. 2021. 'Walking Into a Fire Hoping You Don't Catch': Strategies and Designs to Facilitate Cross-Partisan Online Discussions. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1-30.
- Carlos Ruiz, David Domingo, Josep Lluís Micó, Javier Díaz-Noci, Koldo Meso, and Pere Masip. 2011. Public sphere 2.0? The democratic qualities of citizen debates in online newspapers. The International journal of press/politics, Vol. 16, 4 (2011), 463-487.
- Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 5884-5906.
- Farhana Shahid, Dhruv Agarwal, and Aditya Vashistha. 2025. One Style Does Not Regulate All: Moderation Practices in Public and Private WhatsApp Groups. Proceedings of the ACM on Human-Computer Interaction, Vol. 9, CSCW2 (2025), 1-30.
- John Suler. 2004. The online disinhibition effect. Cyberpsychology & behavior, Vol. 7, 3 (2004), 321-326.
- Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, Vol. 3, 9 (2024), 346.
- Samuel Hardman Taylor, Dominic DiFranzo, Yoon Hyung Choi, Shruti Sannon, and Natalya N. Bazarova. 2019. Accountability and Empathy by Design: Encouraging Bystander Intervention to Cyberbullying on Social Media. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW, Article 118 (nov 2019), 26 pages.
- Michael Henry Tessler, Michiel A. Bakker, Daniel Jarrett, Hannah Sheahan, Martin J. Chadwick, Raphael Koster, Georgina Evans, Lucy Campbell-Gillingham, Tantum Collins, David C. Parkes, Matthew Botvinick, and Christopher Summerfield. 2024. AI can help humans find common ground in democratic deliberation. Science, Vol. 386, 6719 (2024), eadq2852.
- Stella Ting-Toomey, Ge Gao, Paula Trubisky, Zhizhong Yang, Hak Soo Kim, Sung-Ling Lin, and Tsukasa Nishida. 1991. Culture, face maintenance, and styles of handling interpersonal conflict: A study in five cultures. International Journal of conflict management, Vol. 2, 4 (1991), 275-296.
- Yixue Wang and Nicholas Diakopoulos. 2022. Highlighting High-quality Content as a Moderation Strategy: The Role of New York Times Picks in Comment Quality and Engagement. Trans. Soc. Comput., Vol. 4, 4, Article 13 (2022), 24 pages.
- Meng Xia, Qian Zhu, Xingbo Wang, Fei Nie, Huamin Qu, and Xiaojuan Ma. 2022. Persua: A Visual Interactive System to Enhance the Persuasiveness of Arguments in Online Discussion. Proc. ACM Hum.-Comput. Interact., Vol. 6, CSCW2, Article 319 (2022), 30 pages.
- Justine Zhang, Jonathan Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain. 2018. Conversations Gone Awry: Detecting Early Signs of Conversational Failure. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 1350-1361.
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. CoRR, Vol. abs/1904.09675 (2019), 43. · arXiv
Cited by
Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems
Carrera, D., Thomas-Mitchell, J., & Wigdor, D. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791952 · Cited by 3
“Many interfaces resembled commercial instantiations of this archetype, utilizing buttons, dropdown menus, and keywords (in creative writing [51, 58, 70, 80] and other writing contexts [28, 38, 63]).”
Xiao, X., Chen, H., Song, Q., & Yang, Y. (2025) · Science Communication · doi:10.1177/10755470251384848 · Cited by 1
Rauxloh, H. (2026) · Communication Methods and Measures · doi:10.1080/19312458.2026.2704557
Large language models for computational social science
Xu, F., Shao, C., Chen, L., Zeng, Q., Chen, Z., Sukiennik, N., Wang, J., Gao, C., Wang, H., Lian, J., et al. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02539-9 · Cited by 1
Metadata
- Authors
- Shahid, F., Dittgen, M., Naaman, M., & Vashistha, A.
- Affiliations
- Cornell University (US)
- Cornell Tech (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2025-10-16
- Volume
- 9
- Issue
- 7
- Pages
- 1-33
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4415256795
- Cited by
- 3
- References
- 62
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 1.21
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- human-AI collaboration, large language models, constructive disagreement, online comments, Islamophobia, homophobia, comment moderation, politeness, toxic language, AI-generated text
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.