CoChat
Sign Up
Cited by 80Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

Open access
Vu, Tu, Iyyer, Mohit, Wang, Xuezhi, Constant, Noah, Wei, Jerry, Wei, Jason, Tar, Chris, Sung, Yun-Hsuan +3 more·2024·Findings of the Association for Computational Linguistics ACL 2024
DOI 10.18653/v1/2024.findings-acl.813
Open this paper in CoChat — free accountOpen PDF

Abstract

Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong. Findings of the Association for Computational Linguistics: ACL 2024. 2024.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces FRESHQA, a 600-question dynamic QA benchmark for testing LLM factuality on never-changing, slow-changing, fast-changing, and false-premise questions, plus FRESHPROMPT, a few-shot search-engine-augmented prompting method. It finds that offline LLMs struggle badly on fresh knowledge and false premises, while FRESHPROMPT substantially improves accuracy and can make open-source models competitive with commercial search-augmented systems.

Key findings

  • Offline LLMs perform poorly on FRESHQA: overall accuracy ranges from 0.8% to 32.0% under STRICT and 0.8% to 46.4% under RELAXED; all models struggle on fast-changing and false-premise questions, with GPT-4 never exceeding 15% on fast-changing questions in either mode.
  • FRESHPROMPT improves GPT-4 substantially: the best GPT-4 + FRESHPROMPT variant reaches 77.6% STRICT and 79.0% RELAXED overall accuracy, and the paper reports absolute improvements over vanilla GPT-4 of 47% under STRICT and 31.4% under RELAXED in Section 4.3 (the abstract reports 32.6% and 49.0% under RELAXED and STRICT, respectively).
  • FRESHPROMPT outperforms other search-augmented methods: GPT-3.5 + FRESHPROMPT improves over PPLX.AI and SELF-ASK by +3.8% and +14.4% overall under STRICT, respectively; in 2024, MIXTRAL 8X7B + FRESHPROMPT obtains an 11% absolute accuracy improvement over PPLX.AI’s ONLINE LLM 70B.
  • FRESHEVAL, the automatic evaluator, achieves 96.5% average agreement with human judgments for RELAXED and 96% for STRICT.

Methods

  • FRESHQA contains 600 human-written natural questions across four categories (never-changing, slow-changing, fast-changing, false-premise), with one-hop and multi-hop difficulty; it is split into 500 test examples, 100 development examples, and a 15-example demonstration set. Questions were written by NLP researchers and online freelancers, then cleaned and quality-checked.
  • Evaluation uses two modes: RELAXED, which checks only whether the main answer is correct, and STRICT, which additionally requires all claims to be factual and up-to-date. Human evaluation collected >50K judgments; inter-rater agreement was 99% for RELAXED and 96% for STRICT. FRESHEVAL is a few-shot LLM autorater used for later evaluation.
  • FRESHPROMPT queries Google Search with the question verbatim, standardizes retrieved evidences by source, date, title, snippet, and highlighted words, orders them chronologically, and includes few-shot demonstrations with chain-of-thought reasoning. Experiments compare GPT-3.5, GPT-4, SELF-ASK, PPLX.AI, Google Search, and later GPT-4, MIXTRAL 8X7B, LLAMA-2, PPLX.AI’s ONLINE LLM, and YOU.COM’s WEB LLM.

Limitations

  • The paper states that FRESHQA requires regular answer updating, and answers may become stale between updates; FRESHPROMPT uses only one search query per question and could be improved with question decomposition or multiple queries; FRESHQA consists of relatively simple questions, so long-form QA performance is unclear; and FRESHPROMPT relies on in-context learning, so it may underperform fine-tuning approaches.
  • The text also makes evident that FRESHQA is dynamic and questions may be reclassified over time, and that all models were benchmarked in 2023 with some APIs nondeterministic even at temperature 0, posing reproducibility challenges.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-acl.813

Cites

1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 4 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    Open-source LLMs such as MIXTRAL, when combined with FRESHPROMPT, are competitive with closed-source and commercial systems on search-augmented QA.

    “Our latest results with FRESHEVAL suggest that opensource LLMS such as MIXTRAL (Jiang et al., 2024), when combined with FRESHPROMPT, are competitive with closed-source and commercial systems on search-augmented QA.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Jiang et al., 2024 source
  • Findingown data

    FRESHPROMPT outperforms competing search engine-augmented prompting methods such as SELF-ASK and commercial systems such as PERPLEXITY.AI.

    “Our experiments show that FRESH- PROMPT outperforms both competing search engine-augmented prompting methods such as SELF-ASK (Press et al., 2022) as well as commercial systems such as PERPLEXITY.AI.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Press et al., 2022 source
  • Findingown data

    All search-engine augmented approaches (SELF-ASK, PPLX.AI, and FRESHPROMPT) provide substantial improvements over vanilla GPT-3.5 and GPT-4.

    “Overall, all search-engine augmented approaches (SELF-ASK, PPLX.AI, and FRESHPROMPT) provide substantial improvements over vanilla GPT-3.5 and GPT-4.”
    Quote in the paperSubstantial claim
  • Findingown data

    Most LLMs have difficulty with multi-hop questions.

    “Most LLMS have difficulty with multi-hop questions.”
    Quote in the paperSubstantial claim
  • Findingown data

    Removing answer boxes decreases GPT-4 + FRESHPROMPT's overall accuracy by 1.4% under STRICT (1.6% under RELAXED).

    “Removing answer boxes decreases GPT-4 + FRESHPROMPT's overall accuracy by 1.4% under STRICT (1.6% under RELAXED).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 35
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    FRESHPROMPT only performs one search query per question.

    “On the method side, FRESHPROMPT only performs one search query per question, and thus it can be further improved via question decomposition and multiple search queries (Khattab et al., 2022).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Khattab et al., 2022 source

      Section 2.4 (pp. 5–6) describes DSP’s single-query baseline and its multi-hop and multiple-query search strategies, but does not say how many queries FRESHPROMPT performs. A search for the claim about FRESHPROMPT returned only passages about DSP, so the specific one-query claim cannot be verified from this paper. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    FRESHQA is a dynamic QA benchmark of 600 natural questions divided into four categories (never-changing, slow-changing, fast-changing, false-premise) that tests a model's ability to answer questions requiring up-to-date world knowledge.

    “we present FRESHQA, a dynamic QA benchmark that tests a model's ability to answer questions that may require reasoning over up-to-date world knowledge.”
    • Also stated in: § 1 Introduction · plain
      “FRESHQA consists of 600 natural questions that are broadly divided into the four main categories shown in Figure 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The absolute accuracy gap between STRICT and RELAXED diminishes substantially with FRESHPROMPT (from 17.8% to 2.2%), indicating a significant reduction in outdated and hallucinated answers.

    “The absolute accuracy gap between STRICT and RE-LAXED diminishes substantially with FRESHPROMPT (from 17.8% to 2.2%), indicating a significant reduction in outdated and hallucinated answers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    FLAN-PALM and CODEX show minimal hallucination due to their concise and direct answers.

    “In contrast, FLAN-PALM and CODEX show minimal hallucination due to their concise and direct answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    FRESHQA is divided into a test set with 500 examples (125 per question type), a development set with 100 examples (25 per question type), and a 15-example demonstration set for few-shot learning.

    “FRESHQA is divided into a test set with 500 examples (125 per question type), a development set with 100 examples (25 per question type), and a 15-example demonstration set for fewshot learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Increasing the number of demonstrations from 5 to 15 marginally decreases accuracy in both evaluation modes, with a 1% overall decrease under STRICT.

    “Conversely, increasing the number of demonstrations from 5 to 15 marginally decreases accuracy in both evaluation modes, with a 1% overall decrease under STRICT.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    FRESHQA is dynamic: some ground-truth answers may change over time, and questions may be reclassified from one category to another.

    “Additionally, FRESHQA is dynamic in nature: some of the ground-truth answers may change over time, and a question classified under a specific category may undergo reclassification at some later point in time”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Using separate prompts for RELAXED and STRICT evaluations resulted in better performance compared to using a single combined prompt.

    “In our experiments, we found that using separate prompts for RELAXED and STRICT evaluations resulted in better performance compared to using a single, combined prompt for both evaluation modes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 + FRESHPROMPT surpasses all other methods by a substantial margin, with its best variant (15 retrieved evidences per question) achieving 77.6% and 79.0% overall accuracies under STRICT and RELAXED, respectively.

    “GPT-4 + FRESHPROMPT surpasses all other methods by a substantial margin, with its best variant (15 retrieved evidences per question) achieving 77.6% and 79.0% overall accuracies under STRICT and RELAXED, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Adding the premise check boosts accuracy by +23.4% and +6.4% for GPT-3.5 and GPT-4, respectively, under STRICT (+22.6% and +11.3% under RELAXED).

    “For example, adding this premise check boosts accuracy by +23.4% and +6.4% for GPT-3.5 and GPT-4, respectively, under STRICT (+22.6% and +11.3% under RELAXED).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    FRESHEVAL is a reliable autorater for quick evaluation and comparison on FRESHQA.

    “we additionally develop FRESHEVAL, a reliable autorater for quick evaluation and comparison on FRESHQA.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Compared to PPLX.AI and SELF-ASK (all built on top of GPT-3.5), GPT-3.5 + FRESHPROMPT demonstrates a respective increase of +3.8% and +14.4% in overall accuracy under STRICT.

    “Compared to PPLX.AI and SELF-ASK (all built on top of GPT-3.5),GPT-3.5 + FRESHPROMPT demonstrates a respective increase of +3.8% and +14.4% in overall accuracy under STRICT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FRESHEVAL achieves an average agreement of 96.5% with human judgments for RELAXED and 96% for STRICT.

    “achieving an average agreement of 96.5% with human judgments for RELAXED and 96% for STRICT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All models struggle on FRESHQA, with overall accuracy ranging from 0.8% to 32.0% under STRICT and 0.8% to 46.4% under RELAXED.

    “All models struggle on FRESHQA, with overall accuracy ranging from 0.8% to 32.0% under STRICT, and Overall Fast-changing questions False-premise questions Figure 2: Accuracy of different LLMS on FRESHQA under RELAXED and STRICT (no hallucination) evaluations.All models (regardless of model size) fall short on fast-changing and false-premise questions.0.8% to 46.4% under RELAXED.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FRESHPROMPT narrows the performance gap between GPT-4 and MIXTRAL 8X7B to under 9% in all question categories, with the largest gap on slow-changing questions decreasing from 34.4% to 8.8%.

    “FRESHPROMPT narrows the performance gap between GPT-4 and MIXTRAL 8X7B to under 9% in all question categories, with the largest gap on slow-changing questions decreasing from 34.4% to 8.8%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Verbose demonstration answers may aid in tackling intricate questions but can also be detrimental by allowing for hallucination, leading to a 2.6% decrease in overall accuracy under STRICT.

    “Our manual inspection reveals that while verbose answers may aid in tackling intricate questions, they can also be detrimental by allowing for hallucination, leading to a 2.6% decrease in overall accuracy under STRICT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    FRESHPROMPT significantly boosts FRESHQA accuracy: GPT-4 + FRESHPROMPT achieves absolute accuracy improvements of 47% and 31.4% over GPT-4 under STRICT and RELAXED, respectively.

    “FRESHPROMPT offers large improvements over vanilla GPT-3.5 and GPT-4 across the board.GPT-4 + FRESHPROMPT achieves absolute accuracy improvements of 47% and 31.4% over GPT-4 under STRICT and RELAXED, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Larger models do not improve accuracy for T5 and PALM on false-premise questions (flat scaling), with performance ranging from 0.0% to 1.6%.

    “Larger models do not improve accuracy for T5 and PALM (”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All LLMs struggle to answer questions requiring fast-changing world knowledge and questions with false premises that need to be debunked.

    “We observe that all LLMS struggle to answer questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked.”
    • Also stated in: § 1 Introduction · plain
      “for example, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    COT generally outperforms FEW-SHOT under RELAXED, whereas FEW-SHOT performs better under STRICT, as COT introduces more room for hallucination.

    “COT generally outperforms FEW-SHOT under RELAXED, whereas FEW-SHOT performs better under STRICT, as COT introduces more room for hallucination.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 experiences minimal impact from the premise check, with only a 0.6% decrease under STRICT and a 1.2% increase under RELAXED.

    “Conversely, GPT-4 experiences minimal impact, with only a 0.6% decrease under STRICT and a 1.2% increase under RELAXED.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    FRESHQA questions span diverse topics and difficulty levels, requiring single-hop and multi-hop reasoning.

    “FRESHQA's questions span a diverse set of topics with diverse difficulty levels (requiring single-hop and multi-hop reasoning), and require a model to”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Despite its knowledge cutoff date in September 2021, ChatGPT was aware of the recent Russian invasion of Ukraine on February 24, 2022.

    “Despite its knowledge cutoff date in September 2021, CHATGPT was aware of the recent Russian invasion of Ukraine on February 24, 2022.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under STRICT, FEW-SHOT and COT yield +36.1% and +26.9% accuracy improvement over zero-shot prompting with PALM 540B on questions about pre-2022 knowledge (+21.9% and +29.7% under RELAXED).

    “Under STRICT, FEW-SHOT and COT yields +36.1% and +26.9% accuracy improvement, respectively, over zero-shot prompting with PALM 540B on questions about pre-2022 knowledge (+21.9% and +29.7% under RELAXED).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-3.5, ChatGPT, and GPT-4 outperform other models on false-premise questions, achieving accuracies between 25.8% to 42.7% under STRICT and 32.3% to 66.9% under RELAXED.

    “However, GPT-3.5, CHATGPT, and GPT-4 outperform other models significantly, achieving accuracies between 25.8% to 42.7% under STRICT and 32.3% to 66.9% under RE-LAXED.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    T5 Large and XL are incapable of dealing with multi-hop questions, while FLAN-PALM 540B, CODEX, and GPT-3.5 suffer the most when switching from one-hop to multi-hop questions.

    “T5 LARGE and XL are incapable of dealing with this type of questions, while FLAN-PALM 540B, CODEX, and GPT-3.5 suffer the most when switching from one-hop to multi-hop questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Even recent LLMs, with or without search engine augmentation, struggle with questions requiring up-to-date knowledge (accuracies 6.1% to 44.9% on post-2024 knowledge) and false premises (33.1% to 68.5%).

    “As shown in Table 2, even recent LLMS, with or without search engine augmentation, struggle with questions requiring up-to-date knowledge (with accuracies ranging from 6.1% to 44.9% on questions about post-2024 knowledge) as well as questions with false premises (accuracies ranging from 33.1% to 68.5%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 + FRESHPROMPT yields an improvement of 32.6% and 49.0% accuracy over vanilla GPT-4 on FRESHQA under RELAXED and STRICT, respectively.

    “our best GPT-4 + FRESHPROMPT variant yields an improvement of 32.6% and 49.0% accuracy over the vanilla (OpenAI, 2023) on FRESHQA under RELAXED and STRICT, respectively.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites OpenAI, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    The inter-rater agreement between two authors and between FRESHEVAL and each human rater is high across RELAXED, STRICT, and ALL judgements.

    “The high accuracies demonstrate that our evaluation protocol is reproducible and reliable, and FRESHEVAL can be used in place of human evaluation on FRESHQA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under RELAXED, PPLX.AI outperforms GPT-3.5 + FRESHPROMPT by +4.2%, mainly due to its higher accuracy on false-premise questions (58.1% vs. 41.1%).

    “However, under RELAXED, PPLX.AI outperforms GPT-3.5 + FRESHPROMPT by +4.2%, mainly due to its higher accuracy on false-premise questions (58.1% vs. 41.1%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both the number of retrieved evidences and their order are key factors behind the correctness of LLM-generated answers.

    “Further analysis of FRESH-PROMPT reveals that both the number of retrieved evidences and their order are key factors behind the correctness of LLM-generated answers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The high accuracies demonstrate that the evaluation protocol is reproducible and reliable, and FRESHEVAL can be used in place of human evaluation on FRESHQA.

    “Overall, the high accuracies demonstrate that our evaluation protocol is reproducible and reliable, and FRESHEVAL can be used in place of human evaluation on FRESHQA.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    FRESHPROMPT relies on in-context learning and thus may underperform approaches that fine-tune the base LLM on new knowledge.

    “Finally, FRESHPROMPT relies on in-context learning and thus may underperform approaches that fine-tune the base LLM on new knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Each Google Search result contains an associated text snippet along with details, such as source webpage, date, title, and highlighted words.

    “Each result contains an associated text snippet along with details, such as source webpage, date, title, and highlighted words.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The two-mode evaluation procedure uses RELAXED to measure correctness of the main answer and STRICT to measure whether all claims in the response are factual and up-to-date (no hallucination).

    “We benchmark a diverse range of both closed and open-source LLMS under a two-mode evaluation procedure: RELAXED, which measures only whether the main answer is correct; and STRICT, which measures whether all of the claims in the response are factual and up-to-date (i.e., no hallucination).”
    • Also stated in: § 2.1 Evaluation · plain
      “Model responses were evaluated by two authors in a two-mode evaluation procedure: RELAXED, which assesses the correctness of the main answer; and STRICT, which additionally examines whether all of the facts in the answer are accurate (i.e., no hallucination).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Even without access to up-to-date information, a model can still give accurate answers to some current questions by making random guesses or using past valid responses.

    “Note that even without access to up-to-date information, a model can still give accurate answers to some current questions by making random guesses or using past valid responses.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Quality control for FRESHQA included manual review for well-formed questions, removal of duplicates and invalid questions, and verification of answers and supporting URLs.

    “This included manual review for well-formed questions, removal of duplicates and invalid questions (e.g., too easy or controversial), and verification of answers and supporting URLS.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Google Search produces different types of search results for a given query, including the answer box, organic results, knowledge graph, questions and answers from crowdsourced QA platforms, and related questions from search users.

    “GOOGLE SEARCH produces different types of search results for given a query, including the answer box, organic results, and other useful information, such as the knowledge graph, questions and answers from crowdsourced QA platforms, and related questions from search users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Some of the evaluated models may have evolved, posing a challenge to result reproducibility.

    “We note that some of the evaluated models may have evolved, posing a challenge to result reproducibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    FRESHQA needs regular answer updating by maintainers; in the interim between updates, answers to some questions might become stale.

    “One obvious challenge with FRESHQA is the need for regular answer updating by the maintainers; in the interim period between updates, the answers to some questions might become stale.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The development set was reserved for future use and was not used in this paper.

    “The development set is reserved for future use and was not used in this paper.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    FRESHEVAL's evaluation protocol is reproducible and reliable.

    “The high accuracies demonstrate that our evaluation protocol is reproducible and reliable, and FRESHEVAL can be used in place of human evaluation on FRESHQA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Two authors independently evaluated a subset of 100 model responses in both modes and had an agreement of 99% for RELAXED and 96% for STRICT.

    “Two authors independently evaluated a subset of 100 model responses in both modes and had an agreement of 99% for RE-LAXED and 96% for STRICT, validating the reliability of our evaluation protocol.”
    Quote in the paperSubstantial claim
43 of 43

  • The paper relies on this without support.

    Search engine results (Google Search) contain relevant and up-to-date information sufficient to answer FRESHQA questions.

    “FRESHPROMPT uses a text prompt to ( 1) incorporate contextually relevant and current information (including answers to relevant questions) from a search engine into a model, and (2) teach the model to reason over retrieved evidences.”

  • The paper relies on this without support.

    The four question categories (never-changing, slow-changing, fast-changing, false-premise) adequately cover the space of time-sensitive questions relevant to LLM factuality.

    “Our questions are broadly divided into four main categories: never-changing, in which the answer almost never changes; slow-changing, in which the answer typically changes over the course of several years; fast-changing, in which the answer typically changes within a year or less; and false-premise, which includes questions whose premises are factually incorrect and thus have to be rebutted.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported 2024 knowledge accuracy range excludes a table result

    Minor

    The text gives 44.9% as the upper end of the range, but Table 2 reports 59.2% for GPT-4 + FRESHPROMPT in the ≥2024 knowledge column.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume 33, pages 9459-9474.
  2. PMLR.
  3. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.0877.
  4. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023a. Gener- ate rather than retrieve: Large language models are strong context generators. Proceedings of the 11th International Conference on Learning Representa- tions (ICLR 2023).
  5. Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2023c. CREPE: Open-domain question answering with false presuppositions. In Proceedings of the 61st Annual Meeting of the Asso- ciation for Computational Linguistics (ACL), pages 10457-10480.

Cited by

Sorted by citations

Related

Authors
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., & Luong, T.
Affiliations
University of Massachusetts Amherst (US)
ORCID
Venue
Findings of the Association for Computational Linguistics ACL 2024
Publisher
Association for Computational Linguistics
Published
2024
Pages
13697-13720
DOI
10.18653/v1/2024.findings-acl.813
OpenAlex
W4402684046
Cited by
73
References
0
Citations by year
2026: 28 · 2025: 25 · 2024: 20
FWCI
21.44
Type
conference paper
Language
en
Open access
Licence
cc-by
Funders
National Science Foundation (2046248)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, retrieval-augmented generation, search engines, fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review