Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation
Open accessAbstract
In today's digital era, the rapid spread of misinformation poses threats to public well-being and societal trust. As online misinformation proliferates, manual verification by fact checkers becomes increasingly challenging. We introduce FACT-GPT (Fact-checking Augmentation with Claim matching Task-oriented Generative Pre-trained Transformer), a framework designed to automate the claim matching phase of fact-checking using Large Language Models (LLMs). This framework identifies new social media content that either supports or contradicts claims previously debunked by fact-checkers. Our approach employs LLMs to generate a labeled dataset consisting of simulated social media posts. This data set serves as a training ground for fine-tuning more specialized LLMs. We evaluated FACT-GPT on an extensive dataset of social media content related to public health. The results indicate that our fine-tuned LLMs rival the performance of larger pre-trained LLMs in claim matching tasks, aligning closely with human annotations. This study achieves three key milestones: it provides an automated framework for enhanced fact-checking; demonstrates the potential of LLMs to complement human expertise; offers public resources, including datasets and codes, to further research and applications in the fact-checking domain.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have advantages including support for information extraction and synthesis, fact checking, multilingual analysis, and decision-support assistants, but also important limitations in uncertainty handling, self-assessment, and metacognitive reliability in high-stakes settings.
“While acknowledging various advantages of LLMs (Choi and Ferrara, 2024; Zhao et al., 2024), including their support for information extraction and synthesis, facts checking, multilingual analysis, and decision-support assistants, recent work (Griot et al., 2025) has also highlighted their important limitations in uncertainty handling, self-assessment, and metacognitive reliability in high-stakes”
From Enhancement of modern defense capabilities by convergence of cognitive neuroscience and artificial intelligence (2026) · cites this paper as Choi and Ferrara, 2024
Cites
1 retracted
Limited individual attention and online virality of low-quality information
Qiu, X., Oliveira, D. F. M., Shirazi, A. S., Flammini, A., & Menczer, F. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0132 · Cited by 186 · see citing works
RetractedSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
Detecting rumors from microblogs with recurrent neural networks
Ma, J., Gao, W. G., Mitra, P., Kwon, S., Jansen, B. J. J., Wong, K., & Cha, M. (2016) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 1,102 · see citing works
Ferrara, E. (2023) · Sci · doi:10.3390/sci6010003 · Cited by 875 · cited 1× · see citing works
“These results highlight LLMs’ limitations in diverse settings.”
Self-citationChen, E., Lerman, K., & Ferrara, E. (2020) · JMIR Public Health and Surveillance · doi:10.2196/19273 · Cited by 762 · see citing works
Self-citationChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works
Fact-checking: a meta-analysis of what works and for whom
Walter, N., Cohen, J., Holbert, R. L., & Morag, Y. (2020) · Political Communication · doi:10.1080/10584609.2019.1668894 · Cited by 690 · see citing works
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 680 · see citing works
Humans and Technology: Forms of Conjoined Agency in Organizations
Murray, A., Rhymer, J., & Sirmon, D. G. (2021) · Academy of Management Review · doi:10.5465/amr.2019.0186 · Cited by 518 · see citing works
Long-Term Trends in the Public Perception of Artificial Intelligence
Fast, E., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10635 · Cited by 467 · cited 1× · see citing works
“While public sentiment towards AI is generally favorable in news coverage [6, 11], surveys [35], and social media [30], concerns about its misuse for disseminating misinformation exist.”
BackgroundAugGPT: Leveraging ChatGPT for Text Data Augmentation
Dai, H.-X., Liu, Z., Liao, W., Huang, X.-K., Cao, Y.-H., Wu, Z., Zhao, L., Xu, S.-C., Zeng, F., Liu, W., et al. (2023) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536934 · Cited by 322 · see citing works
Ferrara, E. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00250-1 · Cited by 296 · see citing works
Self-citationTranscending Boundaries: Improvisation and Disability in Dance
Whatley, S. (2019) · Pure (Coventry University) · doi:10.1093/oxfordhb · Cited by 265 · see citing works
Huang, F., Kwak, H., & An, J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587368 · Cited by 247 · see citing works
Adapting computational text analysis to social science (and vice versa)
DiMaggio, P. (2015) · Big Data & Society · doi:10.1177/2053951715602908 · Cited by 231 · see citing works
Automated Fact-Checking for Assisting Human Fact-Checkers
Nakov, P., Corney, D. P. A., Hasanain, M., Alam, F., Elsayed, T., Barrón‐Cedeño, A., Papotti, P., Shaar, S., & Martino, G. D. S. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/619 · Cited by 231 · see citing works
Framing Artificial Intelligence in American Newspapers
Chuan, C., Tsai, W. S., & Cho, S. Y. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314285 · Cited by 216 · see citing works
Hassan, N., Zhang, G., Arslan, F., Caraballo, J., Jimenez, D., Gawsane, S., Hasan, S., Joseph, M., Kulkarni, A., Nayak, A. K., et al. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137765.3137815 · Cited by 208 · see citing works
Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing TasksPreprint
Zhu, Y.-M., Zhang, P., Haq, E.-U., Hui, P., & Tyson, G. (2023) · arXiv.org · doi:10.48550/arxiv.2304.10145 · Cited by 183 · see citing works
Checking how fact-checkers check
Lim, C. (2018) · Research & Politics · doi:10.1177/2053168018786848 · Cited by 166 · see citing works
Törnberg, P. (2023) · arXiv · doi:10.48550/arxiv.2304.06588 · Cited by 152 · see citing works
Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary RemarkPreprint
Reiss, M. (2023) · doi:10.31219/osf.io/rvy5p · Cited by 149 · see citing works
Fact-Checking as Idea and Practice in Journalism
Graves, L., & Amazeen, M. A. (2019) · Oxford Research Encyclopedia of Communication · doi:10.1093/acrefore/9780190228613.013.808 · Cited by 144 · see citing works
Public understanding of artificial intelligence through entertainment media
Nader, K., Toprac, P., Scott, S., & Baker, S. (2022) · AI & Society · doi:10.1007/s00146-022-01427-w · Cited by 143 · see citing works
That is a Known Lie: Detecting Previously Fact-Checked Claims
Shaar, S., Babulkov, N., Martino, G. D. S., & Nakov, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.332 · Cited by 137 · see citing works
Social Media Polarization and Echo Chambers in the Context of COVID-19: Case Study
Jiang, J., Ren, X., & Ferrara, E. (2021) · JMIRx Med · doi:10.2196/29570 · Cited by 135 · see citing works
Self-citationShould ChatGPT Be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · doi:10.2139/ssrn.4627814 · Cited by 133 · see citing works
Self-citationChatgpt: a meta-analysis after 2.5 months
Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., & Eger, S. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100541 · Cited by 115 · see citing works
COVID-19 misinformation and the 2020 U.S. presidential election
Chen, E., Chang, H.-C. H., Rao, A., Lerman, K., Cowan, G., & Ferrara, E. (2021) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-57 · Cited by 98 · see citing works
Self-citationSocial bot detection in the age of ChatGPT: Challenges and opportunities
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i6.13185 · Cited by 84 · see citing works
Self-citationRao, A., Morstatter, F., Hu, M., Chen, E., Burghardt, K., Ferrara, E., & Lerman, K. (2021) · Journal of Medical Internet Research · doi:10.2196/26692 · Cited by 80 · see citing works
Self-citationBentivogli, L., Raffaella Bernardi, B., Marelli, M., Menini, S., Baroni, M., & Zamparelli, R. (2016) · Language Resources and Evaluation · doi:10.1007/s10579-015-9332-5 · Cited by 53 · see citing works
He, B., Ahamad, M., & Kumar, S. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583388 · Cited by 49 · see citing works
Nonprobability Sampling and Twitter
Rafail, P. (2017) · Social Science Computer Review · doi:10.1177/0894439317709431 · Cited by 46 · see citing works
CheckThat! at CLEF 2019: Automatic Identification and Verification of Claims
Elsayed, T., Nakov, P., Barrón‐Cedeño, A., Hasanain, M., Suwaileh, R., Martino, G. D. S., & Atanasova, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15719-7_41 · Cited by 43 · see citing works
Factuality Challenges in the Era of Large Language Models
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.05189 · Cited by 32 · see citing works
Self-citationSharma, K., Ferrara, E., & Liu, Y. (2022) · The Web Conference · doi:10.1145/3485447.3512271 · Cited by 13 · see citing works
Self-citationInformation verification improvement by textual entailment methods
Yavary, A., Sajedi, H., & Abadeh, M. S. (2019) · SN Applied Sciences · doi:10.1007/s42452-019-1073-4 · Cited by 5 · see citing works
Flagged first, then by citations
2 of the 43 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Brohi, S., Mastoi, Q.-U.-A., Jhanjhi, N. Z., & Pillai, T. R. (2025) · Algorithms · doi:10.3390/a18080499 · Cited by 69 · see citing works
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48 · see citing works
Silver Lining in the Fake News Cloud: Can Large Language Models Help Detect Misinformation?
Kumar, R., Goddu, B., Saha, S., & Jatowt, A. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3440248 · Cited by 32 · see citing works
Content moderation by LLM: from accuracy to legitimacy
Huang, T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11328-1 · Cited by 24 · see citing works
Assessing the accuracy and explainability of using ChatGPT to evaluate the quality of health news
Liu, X., He, L., Alanazi, E. M., Liu, E., Goss, A., & Gumireddy, L. (2025) · BMC Public Health · doi:10.1186/s12889-025-23206-0 · Cited by 8 · see citing works
Designing Accessible Audio Nudges for Voice Interfaces
Jamshed, H., Nurain, N., & Brewer, R. N. (2025) · International Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713563 · Cited by 8 · see citing works
Emotionally based strategic communications as a new tool in defensive cognitive warfare
Ćosić, K., Popović, S., & Wiederhold, B. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1751406 · Cited by 4 · see citing works
Cosic, K., Kuljanac, A., & Palermo, S. (2026) · Frontiers in psychology · doi:10.3389/fpsyg.2026.1821242 · Cited by 3 · see citing works
“While acknowledging various advantages of LLMs (Choi and Ferrara, 2024; Zhao et al., 2024), including their support for information extraction and synthesis, facts checking, multilingual analysis, and decision-support assistants, recent work (Griot et al., 2025) has also highlighted their important…”
InfluentialFact-Checkers Navigating Generative AI: Practices, Boundaries, and Design Implications
Bozkurt, B., Mosleh, M., & Margetts, H. Z. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806474 · Cited by 1 · see citing works
Improving the Robustness of Rumor Detection Models with Metadata-Augmented Evasive Rumor Datasets
Huynh, L., Gansemer, A., Kim, H., & Hong, J. B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0576-7_25
Oh, Y. J., Rasul, M. E., McKinley, E., & Calabrese, C. R. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1602984
Human vs. Artificial: Detecting Fake News and Disinformation
Hossová, M. P. (2024) · Media & Marketing Identity · doi:10.34135/mmidentity-2024-59
Language Models for Disinformation Detection: To be Large or to be Small?
Kalamakis, I., Liappi, S., Papakostas, D., & Katsaros, D. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315272
Yap, S. C., & Low, Y.-C. (2025) · IEEE International Conference on Computing (ICOCO) · doi:10.1109/icoco67189.2025.11334090
Elgammal, Z., & Alhajj, R. (2026) · Social Science Computer Review · doi:10.1177/08944393261432625
Beyond Accuracy: Experts See AI Fact-Checks as Accurate but Less Useful
Jia, C., Gondimalla, A., Zhang, A., Mullings, D. J., Boltz, A., & Lee, M. K. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791625 · Cited by 1 · see citing works
Multi-Agent System for Orchestrating Retrieval and Multimodal Reasoning in Cloud Environments
Yang, S., Zheng, X., Xu, J., Li, J., Liu, Z., & Ngai, E. C. ‐. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571499
Sword and Shield: Uses and Strategies of LLMs in Navigating Disinformation
Lim, G., Tan, B. C. Z., Sim, K., Shi, W., Chew, M. H., Hee, M. S., Lee, R. K.-W., Perrault, S. T., & Choo, K. T. W. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3817008
- Authors
- Choi, E. C., & Ferrara, E.
- Affiliations
- University of Southern California (US)
- Venue
- Companion Proceedings of the ACM Web Conference 2024
- Publisher
- ACM
- Published
- 2024-05-13
- Pages
- 1441-1449
- OpenAlex
- W4396843602
- Cited by
- 18
- References
- 43
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 3
- FWCI
- 23.61
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International World Wide Web Conference (WWW) (publishedVersion, cc-by)
- Funders
- DARPA (HR001121C0169)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- automated fact-checking, misinformation, large language models, social media content, public health misinformation, fact-checking, LLM fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
