FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Open accessAbstract
Most large language models (LLMs) are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality of LLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Through human evaluations involving more than 50K judgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search engine into the prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that both the number of retrieved evidences and their order play a key role in influencing the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we release FreshQA at github.com/freshllms/freshqa and commit to updating it at regular intervals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FRESHQA was introduced to assess ability of LLMs in dynamic QA about current world knowledge.
“To assess the ability of LLMs in dynamic QA about current world knowledge, Vu et al. [198] introduced FRESHQA.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [198]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143 · see citing works
Flagged first, then by citations
Research works printed but not linked: 35
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research (PMLR) , pp. 2206-2240. PMLR, 2022. URL https://proceedings.mlr.press/ v162/borgeaud22a.html .
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pp. 1877-1901, 2020. URL https://proceedings. neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021a. URL https://arxiv. org/abs/2107.03374 . · arXiv
- Wenhu Chen, Xinyi Wang, William Yang Wang, and William Yang Wang. A dataset for answering time-sensitive questions. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) , volume 1, 2021b. URL https://datasets-benchmarks-proceedings.neurips.cc/paper\_files/ paper/2021/file/1f0e3dad99908345f7439f8ffabdffc4-Paper-round2.pdf .
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. URL https: //arxiv.org/abs/2204.02311 . · arXiv
- Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022. URL https://arxiv.org/abs/2210.11416 . · arXiv
- Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 3558-3567, 2019. URL https://aclanthology.org/ P19-1346 .
- Learning (ICML) , volume 119 of Proceedings of Machine Learning Research (PMLR) , pp. 39293938. PMLR, 2020. URL https://proceedings.mlr.press/v119/guu20a.html .
- Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. 2022. URL https://arxiv.org/abs/2208.03299 . · arXiv
- Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 1601-1611, 2017. URL https://aclanthology.org/P17-1147 .
- Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What's the answer right now? 2022. URL https://arxiv.org/abs/2207.13332 . · arXiv
- Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. 2022. URL https://arxiv.org/abs/2212.14024 . · arXiv
- Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. (QA) 2 : Question answering with questionable assumptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 8466-8487, 2023. URL https://aclanthology.org/ 2023.acl-long.472 .
- Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internetaugmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 , 2022. · arXiv
- Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 3045-3059, November 2021. URL https://aclanthology.org/ 2021.emnlp-main.243 .
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pp. 94599474, 2020. URL https://proceedings.neurips.cc/paper\_files/paper/2020/file/ 6b493230205f780e1bc26945df7481e5-Paper.pdf .
- Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen GilsenanMcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models. In Proceedings of the 39th International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research (PMLR) , pp. 13604-13622. PMLR, 2022. URL https://proceedings.mlr. press/v162/liska22a.html .
- Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 , 2023a. URL https://arxiv.org/abs/2307.03172 . · arXiv
- Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 , 2023. URL https://arxiv. org/abs/2301.13688 . · arXiv
- Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. 2023. URL https://arxiv.org/abs/2305.15225 . · arXiv
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 1906-1919, 2020. URL https://aclanthology.org/ 2020.acl-main.173 .
- Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147 , 2022. URL https://arxiv.org/abs/2203.11147 . · arXiv
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. 2021. URL https://arxiv.org/abs/2112.09332 . · arXiv
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 , 2023. URL https://arxiv.org/ abs/2303.0877 .
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pp. 27730-27744, 2022. URL https://proceedings.neurips.cc/paper\_files/paper/2022/ file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf .
- Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350 , 2022. URL https://arxiv.org/abs/2210.03350 . · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-totext transformer. Journal of Machine Learning Research (JMLR) , 21(140):1-67, 2020. URL https://jmlr.org/papers/v21/20-074.html .
- Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023. URL https://arxiv.org/abs/2302.04761 . · arXiv
- Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. Proceedings of the 11th International Conference on Learning Representations (ICLR 2023) , 2023. URL https://openreview.net/forum?id=-cqvvvb-NkI .
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. URL https://arxiv.org/abs/2201.11903 . · arXiv
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. 2022. URL https://arxiv.org/ abs/2210.03629 . · arXiv
- Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. Proceedings of the 11th International Conference on Learning Representations (ICLR 2023) , 2023a. URL https://openreview.net/forum?id=fB0hRu9GZUS .
- Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. Ifqa: A dataset for open-domain question answering under counterfactual presuppositions. 2023b. URL https://arxiv.org/ abs/2305.14010 . · arXiv
- Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. CREPE: Open-domain question answering with false presuppositions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 10457-10480, 2023c. URL https: //aclanthology.org/2023.acl-long.583 .
- Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 7371-7387, 2021. URL https://aclanthology.org/2021.emnlp-main.586 .
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
“For example, factual-centred metrics [19, 20, 40, 57] and benchmarks [34, 35, 65] have been proposed to measure and reduce hallucination on specific datasets.”
BackgroundTowards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Calibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2024) · Proceedings of the 56th Annual ACM Symposium on Theory of Computing · doi:10.1145/3618260.3649777 · Cited by 91 · see citing works
Long-form factuality in large language models
Wei, J., Yang, C., Song, X., Lu, Y., Hu, N., Huang, J., Tran, D., Peng, D., Liu, R., Huang, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.18802 · Cited by 36 · see citing works
Self-citationKenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36 · see citing works
VeriScore: Evaluating the factuality of verifiable claims in long-form text generation
Song, Y., Kim, Y., & Iyyer, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.552 · Cited by 15 · see citing works
Self-citationMeasuring short-form factuality in large language models
Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W. (2024) · arXiv · doi:10.48550/arxiv.2411.04368 · Cited by 14 · see citing works
Self-citationRanking Manipulation for Conversational Search Engines
Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S. (2024) · arXiv · doi:10.48550/arxiv.2406.03589 · Cited by 8 · see citing works
“This workflow enables access to a dynamic knowledge base not seen during training, reduces the necessary LLM context length, and mitigates model hallucinations (Vu et al., 2023).”
BackgroundEvaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Hu, Y., Wang, Y., & McAuley, J. (2025) · arXiv · doi:10.48550/arxiv.2507.05257 · Cited by 1 · see citing works
“While promising, LOCOMO still features relatively short conversations ( ∼ 9k), and LongMemEval uses synthetic conversations with limited topical diversity, making the dialogues less realistic and potentially less representative of real-world memory use cases.”
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D. (2025) · arXiv · doi:10.48550/arxiv.2509.07968 · Cited by 1 · see citing works
“…respect to a given context (Jacovi et al., 2025; Rashkin et al., 2022), and retrieval-augmented benchmarks which evaluate the use of search tools to access fresh or real-time information beyond the model’s training data (Kasai et al., 2024; Krishna et al., 2025; Vu et al., 2023; Yang et al., 2024).”
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1 · see citing works
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
Liu, T., Qian, C., Cief, M., He, Y., Dan, D., Aletras, N., & Kazai, G. (2026) · arXiv
“Tools commonly include web search (Vu et al., 2024; Jin et al., 2025), code execution (Chen et al., 2022; Gao et al., 2023), and domain-specific APIs.”
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01062
“Furthermore, most evaluations of search-augmented LLM S focus on short and simple factual queries where top-ranked search results directly answer the question [Vu et al., 2024, Kasai et al., 2023].”
InfluentialSelf-citationBrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Zhou, P., Leon, B., Ying, X., Zhang, C., Shao, Y., Ye, Q., Chong, D., Jin, Z., Xie, C., Cao, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.19314
“While an increasing number of studies Li et al. (2023a); Fernández-Pichel et al. (2024); Fan et al. (2024); Vu et al. (2023); Lai et al. (2025); He et al. (2024); Lai et al. (2024); Xiong et al. (2024) demonstrate that web browsing greatly improves LLM performance on downstream tasks, there remains…”
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.07952
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13 · see citing works
- Authors
- Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., & Luong, T.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.03214
- OpenAlex
- W4387432227
- Cited by
- 7
- References
- 1
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2046248)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, factual accuracy, hallucinations, false premises, debunking, few-shot prompting, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
