Working With AI to Persuade: Examining a Large Language Model's Ability to Generate Pro-Vaccination Messages
Open accessAbstract
Artificial Intelligence (AI) is a transformative force in communication and messaging strategy, with potential to disrupt traditional approaches. Large language models (LLMs), a form of AI, are capable of generating high-quality, humanlike text. We investigate the persuasive quality of AI-generated messages to understand how AI could impact public health messaging. Specifically, through a series of studies designed to characterize and evaluate generative AI in developing public health messages, we analyze COVID-19 pro-vaccination messages generated by GPT-3, a state-of-the-art instantiation of a large language model. Study 1 is a systematic evaluation of GPT-3's ability to generate pro-vaccination messages. Study 2 then observed peoples' perceptions of curated GPT-3-generated messages compared to human-authored messages released by the CDC (Centers for Disease Control and Prevention), finding that GPT-3 messages were perceived as more effective, stronger arguments, and evoked more positive attitudes than CDC messages. Finally, Study 3 assessed the role of source labels on perceived quality, finding that while participants preferred AI-generated messages, they expressed dispreference for messages that were labeled as AI-generated. The results suggest that, with human supervision, AI can be used to create effective public health messages, but that individuals prefer their public health messages to come from human institutions rather than AI sources. We propose best practices for assessing generative outputs of large language models in future social science research and ways health professionals can use AI systems to augment public health messaging.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Users increasingly cannot tell the difference between human writing and LLM writing when presented with text, and between a human interlocutor and a contemporary chatbot when in interaction.
“But users increasingly cannot tell the difference between human writing and LLM writing when presented with text (30–32), and between a human interlocutor or a contemporary chatbot when in interaction (17, 26).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 4× · see citing works
“Study 1 uses manual template engineering for the prompts [54], which involves the creation of intuitive natural language as this is the simplest prompt engineering process and replicates how GPT-3 could be used in practice by communicators without technical expertise.”
BackgroundMethodInfluentialRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationCOVID-19 Vaccine Hesitancy Worldwide: A Concise Systematic Review of Vaccine Acceptance Rates
Sallam, M. (2021) · Vaccines · doi:10.3390/vaccines9020160 · Cited by 2,128 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1× · see citing works
“, [6]), algorithmic transparency and explainability ([32, 51]), information and misinformation [9], and the implications of these systems online and offline (e.”
BackgroundTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899 · see citing works
Persuasive computers: perspectives and research directions
Fogg, B. (1998) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '98 · doi:10.1145/274644.274677 · Cited by 434 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75 · see citing works
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social GoodPreprint
Wang, X., Shi, W., Kim, R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06725 · Cited by 28 · see citing works
Persuasive Natural Language Generation -- A Literature ReviewPreprint
Duerr, S., & Gloor, P. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05786 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 59
- Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate Muslims with violence. Nature Machine Intelligence 3, 6 (2021), 461-463.
- Sam Addison Ankenbauer and Alex Jiahong Lu. 2021. Navigating the "Glimmer of Hope": Challenges and Resilience among US Older Adults in Seeking COVID-19 Vaccination. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing. 10-13.
- Kenneth C Arnold, April M Volzer, and Noah G Madrid. 2021. Generative Models can Help Writers without Writing for Them.. In IUI Workshops.
- Robert A Bell, Matthew S McGlone, and Marko Dragojevic. 2014. Vicious viruses and vigilant vaccines: Effects of linguistic agency assignment in health policy advocacy. Journal of health communication 19, 10 (2014), 1178-1195.
- Anne Bowser, Katie Shilton, Jenny Preece, and Elizabeth Warrick. 2017. Accounting for privacy in citizen science: Ethical research in a context of openness. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 2124-2136.
- Stevie Chancellor, Jessica Annette Pater, Trustin Clear, Eric Gilbert, and Munmun De Choudhury. 2016. # thyghgapp: Instagram content moderation and lexical variation in pro-eating disorder communities. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. 1201-1213.
- Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. 2021. All That's' Human'Is Not Gold: Evaluating Human Evaluation of Generated Text. arXiv preprint arXiv:2107.00061 (2021). · arXiv
- Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta Gonzalez, Brian Keegan, Mor Naaman, and Hanna Wallach. 2014. Computational social science: Cscw in the social media era. In Proceedings of the companion publication of the 17th ACM conference on Computer supported cooperative work & social computing. 105-108.
- Thomas Davenport and Ravi Kalakota. 2019. The potential for artificial intelligence in healthcare. Future healthcare journal 6, 2 (2019), 94.
- Ashok Deb, Anuja Majmundar, Sungyong Seo, Akira Matsui, Rajat Tandon, Shen Yan, Jon-Patrick Allem, and Emilio Ferrara. 2018. Social bots for online public health interventions. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 1-4.
- Michael A Devito, Ashley Marie Walker, Jeremy Birnholtz, Kathryn Ringland, Kathryn Macapagal, Ashley Kraus, Sean Munson, Calvin Liang, and Herman Saksono. 2019. Social technologies for digital wellbeing among marginalized communities. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 449-454.
- Mark Diaz. 2019. Algorithmic Technologies and Underrepresented Populations. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 47-51.
- Amiel A Dror, Netanel Eisenbach, Shahar Taiber, Nicole G Morozov, Matti Mizrachi, Asaf Zigron, Samer Srouji, and Eyal Sela. 2020. Vaccine hesitancy: the next challenge in the fight against COVID-19. European journal of epidemiology 35, 8 (2020), 775-779.
- J Richard Eiser and Camilla J White. 1974. Evaluative consistency and social judgment. Journal of Personality and Social Psychology 30, 3 (1974), 349.
- Pouyan Esmaeilzadeh. 2020. Use of AI-based tools for healthcare purposes: a survey study from consumers' perspectives. BMC medical informatics and decision making 20, 1 (2020), 1-19.
- Matthew C Farrelly, James Nonnemaker, Kevin C Davis, and Altijani Hussin. 2009. The influence of the national truth® campaign on smoking initiation. American journal of preventive medicine 36, 5 (2009), 379-384.
- Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 4 (2020), 681-694.
- Karen Freberg, Kristin Saling, Kathleen G Vidoloff, and Gina Eosco. 2013. Using value modeling to evaluate social media messages: The case of Hurricane Irene. Public Relations Review 39, 3 (2013), 185-192.
- Ilona Fridman, Nicole Lucas, Debra Henke, and Christina K Zigler. 2020. Association between public knowledge about COVID-19, trust in information sources, and adherence to social distancing: cross-sectional survey. JMIR public health and surveillance 6, 3 (2020), e22060.
- Susanne Gaube, Harini Suresh, Martina Raue, Alexander Merritt, Seth J Berkowitz, Eva Lermer, Joseph F Coughlin, John V Guttag, Errol Colak, and Marzyeh Ghassemi. 2021. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine 4, 1 (2021), 1-8.
- Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems. 1-19.
- Ben Green and Yiling Chen. 2019. Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments. In Proceedings of the conference on fairness, accountability, and transparency. 90-99.
- Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? analyzing and predicting convinc- ingness of web arguments using bidirectional lstm. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1589-1599.
- Jeffrey T Hancock, Mor Naaman, and Karen Levy. 2020. AI-mediated communication: definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication 25, 1 (2020), 89-100.
- Will Douglas Heaven. 2020. OpenAI's new language generator GPT-3 is shockingly good-and completely mindless. MIT Technology Review (2020).
- Jess Hohenstein and Malte Jung. 2020. AI as a moral crumple zone: The effects of AI-mediated communication on attribution and trust. Computers in Human Behavior 106 (2020), 106190.
- Joo-Wha Hong and Nathaniel Ming Curran. 2019. Artificial intelligence, artists, and art: attitudes toward artwork produced by humans vs. artificial intelligence. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15, 2s (2019), 1-16.
- Li Huang, Zhi Lu, and Priyali Rajagopal. 2022. Numbers, not lives: Ai dehumanization undermines covid-19 preventive intentions. Journal of the Association for Consumer Research 7, 1 (2022), 63-71.
- Anthony Hunter, Lisa Chalaguine, Tomasz Czernuszenko, Emmanuel Hadoux, and Sylwia Polberg. 2019. Towards computational persuasion via natural language argumentation dialogues. In Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz). Springer, 18-33.
- Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T Hancock, and Mor Naaman. 2019. AI-mediated communication: How the perception that profile text was written by AI affects trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-13.
- Tae Woo Kim and Adam Duhachek. 2020. Artificial intelligence and persuasion: A construal-level account. Psychological science 31, 4 (2020), 363-380.
- Andy J King and Allison J Lazard. 2020. Advancing visual health communication research to improve infodemic response. Health Communication 35, 14 (2020), 1723-1728.
- Mary E Laffidy. 2021. Governments' Use of Fear Appeals and Accessible Language in COVID-19 Outreach: Comparing Florida and New York's Messages, March 2020-September 2020. Ph. D. Dissertation. Northern Arizona University.
- Min Kyung Lee and Su Baykal. 2017. Algorithmic mediation in group decisions: Fairness perceptions of algorithmically mediated vs. discussion-based social division. In Proceedings of the 2017 acm conference on computer supported cooperative work and social computing. 1035-1048.
- Sangwon Lee, Seungahn Nah, Deborah S Chung, and Junghwan Kim. 2020. Predicting ai news credibility: communica- tive or social capital or both? Communication Studies 71, 3 (2020), 428-447.
- Hannah Mieczkowski, Jeffrey T Hancock, Mor Naaman, Malte Jung, and Jess Hohenstein. 2021. AI-Mediated Commu- nication: Language Use and Interpersonal Effects in a Referential Communication Task. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-14.
- Marie-Francine Moens. 2018. Argumentation mining: How can a machine acquire common sense and world knowledge? Argument & Computation 9, 1 (2018), 1-14.
- Xiaoli Nan, Irina A Iles, Bo Yang, and Zexin Ma. 2022. Public health messaging during the COVID-19 pandemic and beyond: Lessons from communication science. Health Communication 37, 1 (2022), 1-19.
- Kimberly H Nguyen, David Yankey, Kelsey C Coy, Kathryn A Brookmeyer, Neetu Abad, Rebecca Guerin, Girija Syamlal, Peng-jun Lu, Brittney N Baack, Hilda Razzaghi, et al. 2021. COVID-19 Vaccination Coverage, Intent, Knowledge, Attitudes, and Beliefs among Essential Workers, United States. Emerging infectious diseases 27, 11 (2021), 2908.
- Jeff Niederdeppe, Matthew C Farrelly, James Nonnemaker, Kevin C Davis, and Lauren Wagner. 2011. Socioeconomic variation in recall and perceived effectiveness of campaign advertisements to promote smoking cessation. Social science & medicine 72, 5 (2011), 773-780.
- Fayika Farhat Nova, Pratyasha Saha, Md Shafiur Raihan Shafi, and Shion Guha. 2019. Sharing Of Public Harassment Experiences on Social Media in Bangladesh. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 324-329.
- Harri Oinas-Kukkonen and Marja Harjumaa. 2009. Persuasive systems design: Key issues, process model, and system features. Communications of the association for Information Systems 24, 1 (2009), 28.
- Kirsten Ostherr. 2020. Artificial intelligence and medical humanities. Journal of Medical Humanities (2020), 1-22.
- Sun Young Park, Pei-Yi Kuo, Andrea Barbarin, Elizabeth Kaziunas, Astrid Chow, Karandeep Singh, Lauren Wilcox, and Walter S Lasecki. 2019. Identifying challenges and opportunities in human-AI collaboration in healthcare. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 506-510.
- Jessica A Pater, Oliver L Haimson, Nazanin Andalibi, and Elizabeth D Mynatt. 2016. "Hunger Hurts but Starving Works" Characterizing the Presentation of Eating Disorders Online. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 1185-1200.
- Jane Pirkis, Alyssia Rossetto, Angela Nicholas, Maria Ftanou, Jo Robinson, and Nicola Reavley. 2019. Suicide prevention media campaigns: a systematic literature review. Health communication 34, 4 (2019), 402-414.
- Melissa Roemmele and Andrew S Gordon. 2018. Automated assistance for creative writing with an rnn language model. In Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion. 1-2.
- Shaunak Sastry and Alessandro Lovari. 2017. Communicating the ontological narrative of Ebola: An emerging disease in the time of "epidemic 2.0". Health communication 32, 3 (2017), 329-338.
- Jakob Schoeffer and Niklas Kuehl. 2021. Appropriate fairness perceptions? On the effectiveness of explanations in enabling people to assess the fairness of automated decision systems. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing. 153-157.
- Bryan C Semaan, Scott P Robertson, Sara Douglas, and Misa Maruyama. 2014. Social media supporting political deliberation across multiple public spheres: towards depolarization. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1409-1421.
- Donghoon Shin, Subeen Park, Esther Hehsun Kim, Soomin Kim, Jinwook Seo, and Hwajung Hong. 2022. Exploring the Effects of AI-assisted Emotional Support Processes in Online Mental Health Community. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1-7.
- Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal 31, 2 (2018), 47-53.
- Michael D Slater and June A Flora. 1991. Health lifestyles: Audience segmentation analysis for public health interven- tions. Health education quarterly 18, 2 (1991), 221-233.
- Hyeonjin Soh. 2006. Measuring trust in advertising. Ph. D. Dissertation. uga.
- Patric R Spence, Chad Edwards, Autumn Edwards, Adam Rainear, and Xianlin Jin. 2021. "They're always wrong anyway": exploring differences of credibility, attraction, and behavioral intentions in professional, amateur, and robotic-delivered weather forecasts. Communication Quarterly 69, 1 (2021), 67-86.
- Marianne Udow-Phillips and Paula M Lantz. 2020. Trust in public health is essential amid the COVID-19 pandemic. Journal of Hospital Medicine 15, 7 (2020), 431-433.
- Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language 67 (2021), 101151.
- Ben Young, Sarah Lewis, Srinivasa Vittal Katikireddi, Linda Bauld, Martine Stead, Kathryn Angus, Mhairi Campbell, Shona Hilton, James Thomas, Kate Hinds, et al. 2018. Effectiveness of mass media campaigns to reduce alcohol consumption and harm: a systematic review. Alcohol and alcoholism 53, 3 (2018), 302-316.
- Xiaoquan Zhao, Andrew Strasser, Joseph N Cappella, Caryn Lerman, and Martin Fishbein. 2011. A measure of perceived argument strength: Reliability and validity. Communication methods and measures 5, 1 (2011), 48-75.
Cited by
Sorted by citations
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570 · see citing works
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369 · see citing works
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · Science · doi:10.1126/science.adq1814 · Cited by 354 · see citing works
The Potential of Generative AI for Personalized Persuasion at Scale
Matz, S., Teeny, J., Vaid, S. S., Peters, H., Harari, G. M., & Cerf, M. (2023) · doi:10.31234/osf.io/rn97c · Cited by 312 · see citing works
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
The potential of generative AI for personalized persuasion at scale
Matz, S. C., Teeny, J. D., Vaid, S. S., Peters, H., Harari, G. M., & Cerf, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-53755-0 · Cited by 315 · see citing works
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265 · see citing works
Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking
Sharma, N., Liao, Q. V., & Xiao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642459 · Cited by 132 · see citing works
Evaluating the persuasive influence of political microtargeting with large language models
Hackenburg, K., & Margetts, H. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403116121 · Cited by 114 · see citing works
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110 · see citing works
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84 · see citing works
The Persuasive Power of Large Language Models
Breum, S. M., Egdal, D. V., Gram Mortensen, V., Møller, A. G., & Aiello, L. M. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31304 · Cited by 57 · see citing works
Developing Personalized Marketing Service Using Generative AI
Lee, G. H., Lee, K. J., Jeong, B., & Kim, T. (2024) · IEEE Access · doi:10.1109/access.2024.3361946 · Cited by 44 · see citing works
'It was 80% me, 20% AI': Seeking Authenticity in Co-Writing with Large Language Models
Hwang, A. H.-C., Liao, Q. V., Blodgett, S. L., Olteanu, A., & Trischler, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711020 · Cited by 42 · see citing works
Danry, V., Pataranutaporn, P., Groh, M., & Epstein, Z. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713408 · Cited by 35 · see citing works
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · doi:10.31234/osf.io/xcwdn · Cited by 28 · see citing works
Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trust
Sun, Y., & Wang, T. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791079 · Cited by 16
“Our findings highlight the emerging risks associated with the increasing trend of humanizing LLMs and contribute to the expanding research on AI persuasion [38, 92] and trust calibration in human-AI interactions [44, 91].”
Evaluating Frontier Models for Dangerous Capabilities
Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.13793 · Cited by 11 · see citing works
The Health Communications Assessment Checklist: a teaching tool for uncovering bias in the age of AI
Ofrane, R. H., Canada, T., & Birnbaum, A. S. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1872506
- Authors
- Karinshak, E., Liu, S. X., Park, J. S., & Hancock, J. T.
- Affiliations
- University of Georgia (US)
- Stanford University, Stanford, CA, USA
- Stanford University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-04-14
- Volume
- 7
- Issue
- CSCW1
- Pages
- 1-29
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4365999098
- Cited by
- 167
- References
- 14
- Citations by year
- 2026: 46 · 2025: 76 · 2024: 40 · 2023: 5
- FWCI
- 45.43
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- NSF (AW940028)
- Topics
- Social Sciences › Health › Vaccine Coverage and Hesitancy
- Social Sciences › Communication › Communication and COVID-19 Impact
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Media Influence and Health
- Keywords
- large language models, GPT-3, persuasive communication, COVID-19 vaccination, public health communication, source credibility, human-AI comparison, attitudes toward AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
