Accuracy of a large language model in distinguishing anti- and pro-vaccination messages on social media: The case of human papillomavirus vaccination
Open accessAbstract
Objective: Vaccination has engendered a spectrum of public opinions, with social media acting as a crucial platform for health-related discussions. The emergence of artificial intelligence technologies, such as large language models (LLMs), offers a novel opportunity to efficiently investigate public discourses. This research assesses the accuracy of ChatGPT, a widely used and freely available service built upon an LLM, for sentiment analysis to discern different stances toward Human Papillomavirus (HPV) vaccination. Methods: Messages related to HPV vaccination were collected from social media supporting different message formats: Facebook (long format) and Twitter (short format). A selection of 1,000 human-evaluated messages was input into the LLM, which generated multiple response instances containing its classification results. Accuracy was measured for each message as the level of concurrence between human and machine decisions, ranging between 0 and 1. Results: = .029) for anti- and pro-vaccination short-form, respectively. Using only three or even one instance did not lead to a severe decrease in accuracy. However, for long-form messages, the language model exhibited significantly lower accuracy in categorizing pro-vaccination messages than anti-vaccination ones. Conclusions: ChatGPT shows potential in analyzing public opinions on HPV vaccination using social media content. However, understanding the characteristics and limitations of a language model within specific public health contexts remains imperative.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models (LLMs) can be integrated to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks.
“Moreover, large language models (LLMs) can be integrated into this principle to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks [23, 48–52].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Attitudes to vaccination: A critical review
Yaqub, O., Castle-Clarke, S., Sevdalis, N., & Chataway, J. (2014) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.04.018 · Cited by 1,089 · see citing works
Social media use in the United States: implications for health communication
Chou, W.-Y. S., Hunt, Y. M., Beckjord, E. B., Moser, R. P., & Hesse, B. W. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1249 · Cited by 1,082 · see citing works
Paul, E., Steptoe, A., & Fancourt, D. E. (2020) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2020.100012 · Cited by 1,014 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2023) · doi:10.2139/ssrn.4389233 · Cited by 422 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
Trust in science, social consensus and vaccine confidence
Sturgis, P., Brunton‐Smith, I., & Jackson, J. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01115-7 · Cited by 318 · see citing works
Where We Go From Here: Health Misinformation on Social Media
Chou, W. S., Gaysynsky, A., & Cappella, J. N. (2020) · American Journal of Public Health · doi:10.2105/ajph.2020.305905 · Cited by 311 · see citing works
A cross-sectional examination of marketing of electronic cigarettes on Twitter
Huang, J., Kornfield, R., Szczypka, G., & Emery, S. (2014) · Tobacco Control · doi:10.1136/tobaccocontrol-2014-051551 · Cited by 271 · see citing works
Dunn, A. G., Surian, D., Leask, J., Dey, A., Mandl, K. D., & Coiera, E. W. (2017) · Vaccine · doi:10.1016/j.vaccine.2017.04.060 · Cited by 264 · see citing works
Large language models as a substitute for human experts in annotating political text
Heseltine, M., & von Hohenberg, B. C. (2024) · Research & Politics · doi:10.1177/20531680241236239 · Cited by 116 · see citing works
Sonawane, K., Lin, Y.-Y., Damgacioglu, H., Zhu, Y., Fernández, M. E., Montealegre, J. R., Cazaban, C. G., Li, R., Lairson, D. R., Lin, Y. C., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.24502 · Cited by 102 · see citing works
Evaluating the logical reasoning ability of chatgpt and GPT-4
Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.03439 · Cited by 100 · see citing works
Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., & Cambria, E. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3536008 · Cited by 76 · see citing works
Shapiro, G. K., Surian, D., Dunn, A. G., Perry, R., & Kelaher, M. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016869 · Cited by 64 · see citing works
Cross-platform social dynamics: an analysis of ChatGPT and COVID-19 vaccine conversations
Alipour, S., Galeazzi, A., Sangiorgio, E., Avalle, M., Bojić, L., Cinelli, M., & Quattrociocchi, W. (2024) · Scientific Reports · doi:10.1038/s41598-024-53124-x · Cited by 27 · see citing works
Cruickshank, I. J., Ginossar, T., Sulskis, J., Zheleva, E., & Berger-Wolf, T. Y. (2021) · Journal of Medical Internet Research · doi:10.2196/29127 · Cited by 20 · see citing works
Hornik, R., Binns, S., Emery, S., Epstein, V. M., Jeong, M., Kim, K., Kim, Y., Kranzler, E. C., Jesch, E., Lee, S. J., et al. (2022) · Journal of Communication · doi:10.1093/joc/jqab052 · Cited by 18 · see citing works
Self-citationValdez, D., Soto-Vásquez, A. D., & Montenegro, M. (2023) · Social Science & Medicine · doi:10.1016/j.socscimed.2023.116365 · Cited by 18 · see citing works
Human Papillomavirus Vaccine Impact on Cervical Precancers in a Low-Vaccination Population
Shing, J. Z., Griffin, M. R., Chang, R. S.-W., Beeghly-Fadiel, A. C., Sudenga, S. L., Slaughter, J. C., Pemmaraju, M., Mitchel, E. F., & Hull, P. C. (2021) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2021.08.017 · Cited by 8 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514
Flagged first, then by citations
Cited by
Sorted by citations
Chen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
Multitask Learning for Crash Analysis: A Fine-Tuned LLM Framework Using Twitter Data
Jaradat, S., Nayak, R., Paz, A., Ashqar, H. I., & Elhenawy, M. M. Z. (2024) · Smart Cities · doi:10.3390/smartcities7050095 · Cited by 45 · see citing works
Large Language Models for Health Care Text Classification: Systematic Review
Sakai, H., & Lam, S. S. (2025) · JMIR AI · doi:10.2196/79202 · Cited by 43 · see citing works
Chen, X., Wen-xin, M., Li, D., Zhu, F., Routray, S., Guduri, M., & Margala, M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3527340 · Cited by 17 · see citing works
A suite of large language models for public health infoveillance
Zhou, X., Zhou, J., Wang, C., Xie, Q., Ding, K., Mao, C., Liu, Y., Cao, Z., Chu, H., Chen, X., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02435-6 · Cited by 14 · see citing works
A comparison of the persuasiveness of human and ChatGPT generated pro-vaccine messages for HPV
Xia, D., Song, M., & Zhu, T. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1515871 · Cited by 8 · see citing works
Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review
Sebo, P. (2024) · Journal of Medical Internet Research · doi:10.2196/57667 · Cited by 7 · see citing works
Bragazzi, N. L., Buchinger, M., Atwan, H., Tuma, R., Chirico, F., Szarpak, Ł., Farah, R. E., & Khamisy‐Farah, R. (2025) · JMIR Formative Research · doi:10.2196/56126 · Cited by 5 · see citing works
Kim, S., Kim, K., & Kim, H. M. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.118328 · Cited by 2 · see citing works
Self-citationVaccine Hesitancy on YouTube: A Competition Between Health and Politics
Mejova, Y., & Tizzani, M. (2025) · International Digital Public Health Conference (DPH) · doi:10.1109/dph66411.2025.11198076 · Cited by 2 · see citing works
Sentiment analysis of research attention: the Altmetric proof of concept
Areia, C. M., Taylor, M., Garcia, M., & Hernandez, J. M. (2025) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2025.1612216 · Cited by 2 · see citing works
Khan, S. H., Williams, M. S., & Lybarger, K. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001275 · Cited by 1 · see citing works
Xu, X., Mu, Y., Xue, J., & Matouš, P. (2026) · ENGINEERING Management · doi:10.1007/s42524-026-4186-7 · Cited by 1 · see citing works
MetaHarm: Harmful YouTube Video Dataset Annotated by Domain Experts, GPT-4-Turbo, and Crowdworkers
Jo, W., & Wojcieszak, M. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35951
Liu, X., Liao, Y., Feng, F., Tu, Y., Kang, X., Liu, M., & Han, N. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1826042
Praschk, A., Fischill-Neudeck, V., Caspari, T., & Wiesinger, H. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02449-3
Jo, C. W., & Wojcieszak, M. (2026) · Journal of Communication · doi:10.1093/joc/jqag014
Khanam, F., & Borah, P. (2026) · Atlantic Journal of Communication · doi:10.1080/15456870.2026.2754507
- Authors
- Kim, S., Kim, K., & Wonjeong Jo, C.
- Affiliations
- University of California, Davis (US)
- Cornell University (US)
- Corresponding author
- Kim, Soojong
- Venue
- Preventive Medicine Reports
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 42
- Pages
- 102723
- ISSN
- 2211-3355 (ISSN-L)
- PMC
- PMC11039308
- OpenAlex
- W4394934956
- Cited by
- 18
- References
- 22
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 3
- FWCI
- 2.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Preventive Medicine Reports (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › Health › Vaccine Coverage and Hesitancy
- Keywords
- large language models, ChatGPT, sentiment analysis, HPV vaccination, social media discourse, public opinion on vaccination, Twitter, Facebook
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
