Web Retrieval Agents for Evidence-Based Misinformation Detection
Open accessAbstract
This paper develops an agent-based automated fact-checking approach for detecting misinformation. We demonstrate that combining a powerful LLM agent, which does not have access to the internet for searches, with an online web search agent yields better results than when each tool is used independently. Our approach is robust across multiple models, outperforming alternatives and increasing the macro F1 of misinformation detection by as much as 20 percent compared to LLMs without search. We also conduct extensive analyses on the sources our system leverages and their biases, decisions in the construction of the system like the search tool and the knowledge base, the type of evidence needed and its impact on the results, and other parts of the overall process. By combining strong performance with in-depth understanding, we hope to provide building blocks for future search-enabled misinformation mitigation systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Early works have applied web search agents to fact-checking settings, demonstrating their effectiveness in evidence-based misinformation detection.
“Early works have further applied web search agents to fact-checking settings, demonstrating their effectiveness in evidence-based misinformation detection [24].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 31
- Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. A review on fact extraction and verification. ACM Computing Surveys (CSUR), 55(1):1-35, 2021.
- Canyu Chen and Kai Shu. Combating misinformation in the age of llms: Opportunities and challenges. arXiv preprint arXiv:2311.05656, 2023. · arXiv
- I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. Factool: Factuality detection in generative ai-a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307.13528, 2023. · arXiv
- Tsun-Hin Cheung and Kin-Man Lam. Factllama: Optimizing instruction-following language models with external knowledge for automated fact-checking. In 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 846-853. IEEE, 2023.
- Nima Ebadi, Mohsen Jozani, Kim-Kwang Raymond Choo, and Paul Rad. A memory network information retrieval model for identification of news misinformation. IEEE Transactions on Big Data, 8(5):1358-1370, 2021.
- Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-Augmented Generation for Large Language Models: A Survey. Technical report, January 2024. URL http://arxiv.org/abs/2312. 10997. arXiv:2312.10997 [cs] type: article. · arXiv
- Ashim Gupta and Vivek Srikumar. X-FACT: A New Benchmark Dataset for Multilingual Fact Checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Online, July 2021. Association for Computational Linguistics.
- Yi-Li Hsu, Shih-Chieh Dai, Aiping Xiong, and Lun-Wei Ku. Is explanation the cure? misinformation mitigation in the short term and long term. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1313-1323, 2023.
- Medeswara Rao Kondamudi, Somya Ranjan Sahoo, Lokesh Chouhan, and Nandakishor Yadav. A comprehensive survey of fake news in social networks: Attributes, features, and detection approaches. Journal of King Saud University-Computer and Information Sciences, 35(6):101571, 2023.
- Hao Liao, Jiahao Peng, Zhanyi Huang, Wei Zhang, Guanghua Li, Kai Shu, and Xing Xie. Muser: A multi-step evidence retrieval enhancement framework for fake news detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4461-4472, 2023.
- Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022. · arXiv
- Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023. · arXiv
- Yury Orlovskiy, Camille Thibault, Anne Imouza, Jean-Franc ¸ois Godbout, Reihaneh Rabbany, and Kellin Pelrine. Uncertainty resolution in misinformation detection. arXiv preprint arXiv:2401.01197, 2024. · arXiv
- Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. FaVIQ: Fact verification from information seeking questions. In ACL, 2022.
- Kellin Pelrine, Meilina Reksoprodjo, Caleb Gupta, Joel Christoph, and Reihaneh Rabbany. To- wards reliable misinformation mitigation: Generalization, uncertainty, and gpt-4. arXiv preprint arXiv:2305.14928, 2023. · arXiv
- Mauricio Rivera, Jean-Franc ¸ois Godbout, Reihaneh Rabbany, and Kellin Pelrine. Combining con- fidence elicitation and sample-based methods for uncertainty quantification in misinformation mitigation. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pp. 114-126, 2024.
- Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. Technical report, November 2023. URL http://arxiv.org/abs/2311.09476. arXiv:2311.09476 [cs] type: article. · arXiv
- Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2387-2413, 2023.
- Megha Sundriyal, Ganeshan Malhotra, Md Shad Akhtar, Shubhashis Sengupta, Andrew Fano, and Tanmoy Chakraborty. Document retrieval and claim verification to mitigate covid-19 misinforma- tion. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, pp. 66-74, 2022.
- Petter Törnberg. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588, 2023. · arXiv
- Haoran Wang and Kai Shu. Explainable claim verification via knowledge-grounded reasoning with large language models. arXiv preprint arXiv:2310.05253, 2023. · arXiv
- Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Large Search Model: Redefining Search Stack in the Era of LLMs. Technical report, January 2024. URL http://arxiv.org/abs/2310.14587. arXiv:2310.14587 [cs] type: article. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
- Kai-Cheng Yang and Filippo Menczer. Large language models can rate news outlet credibility. arXiv preprint arXiv:2304.00228, 2023. · arXiv
- Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. · arXiv
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. · arXiv
- Long Ying, HUI Yu, Jinguang Wang, Yongze Ji, and Shengsheng Qian. Fake news detection via multi-modal topic memory network. IEEE Access, 9:132818-132829, 2021.
- Hao Yu, Zachary Yang, Kellin Pelrine, Jean Francois Godbout, and Reihaneh Rabbany. Open, closed, or small language models for text classification? arXiv preprint arXiv:2308.10092, 2023. · arXiv
- Congzhi Zhang, Linhai Zhang, and Deyu Zhou. Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19533-19541, 2024.
- Xuan Zhang and Wei Gao. Towards llm-based fact verification on news claims with a hierarchical step-by-step prompting method. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 996-1011, 2023.
- Xinyi Zhou and Reza Zafarani. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Computing Surveys (CSUR), 53(5):1-40, 2020.
Cited by
Sorted by citations
Linguistic features of AI mis/disinformation and the detection limits of LLMs
Ma, Y., Zhang, X., Ren, J., Wang, R., Wang, M., & Chen, Y. (2025) · Nature Communications · doi:10.1038/s41467-025-67145-1 · Cited by 10 · see citing works
Misinformation and Disinformation in Generative AI—A Survey
Srinivasan, R. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8183-9_23 · Cited by 5 · see citing works
Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection
Li, M., Lin, Z., & Ma, S. (2026) · arXiv · doi:10.48550/arxiv.2605.08583
“The final stage, Web Search , uses the Web Agent again, but now with a search query generated from the citation record rather than a direct URL, in the spirit of multi-agent systems that collect evidence from open-web sources for misinformation detection and structured data acquisition [28, 16].”
Shi, K., Sun, W., Zhang, Z., Sun, L., Chawla, N. V., & Ye, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.23452
- Authors
- Tian, J.-J., Yu, H., Orlovskiy, Y., Vergho, T., Rivera, M., Goel, M., Yang, Z., Godbout, J.-F., Rabbany, R., & Pelrine, K.
- Affiliations
- Dartmouth College (US)
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2409.00009
- OpenAlex
- W4402952906
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- Institut de Valorisation des Données
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- misinformation detection, fact-checking, LLM agents, web search agents, multi-agent systems, search bias, knowledge base construction, evidence requirements, misinformation mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
