Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

In the paper

“Like T5 [51] or BART, RAG can be fine-tuned on any seq2seq task, whereby both the generator and retriever are jointly learned.”

What the sources say

• Supported

  The citation identifies T5 as an example of a model that can be fine-tuned for text-to-text tasks. Section 2.4 (pp. 8–9) describes a common fine-tuning framework for diverse tasks and says the model can be fine-tuned separately on individual tasks. The statement about jointly learning RAG’s generator and retriever is not attributed to the T5 citation. (LLM, conf=0.90)

  cites [51] [51] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints , 2019. URL https://arxiv.org/abs/1910.10683 . - source
• Supported

  The citation identifies BART as the seq2seq model used for comparison, not as evidence about RAG’s retriever. Section 5.3 (Generation Tasks; page unavailable) says BART is fine-tuned as a standard sequence-to-sequence model, and §3.3 gives examples of generation tasks. (LLM, conf=0.90)

  cites [32] [32] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019. URL https://arxiv.org/abs/1910.13461 . - source

In the paper

“REALM [20] and ORQA [31], two recently introduced models that combine masked language models [8] with a differentiable retriever, have shown promising results, but have only explored open-domain extractive question answering.”

What the sources say

• Supported

  The Abstract (p. 1) describes REALM using masked-language-model training with backpropagation through retrieval, and the Introduction (p. 2) reports strong results on three open-domain QA benchmarks. Section 4.2 (p. 6) describes the evaluated retrieval-based approach as extracting answers from documents and identifies ORQA as a closely related learned-retrieval model. The paper’s reported downstream experiments are in open-domain QA. (LLM, conf=0.88)

  cites [20] [20] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. ArXiv , abs/2002.08909, 2020. URL https: //arxiv.org/abs/2002.08909 . - source
• Supported

  For ORQA, the cited paper’s Section 3 says its jointly learned retriever and reader use BERT-based scoring components. The Retriever component section describes a span-based reader, and the Introduction reports improvements on open-domain QA datasets. This citation supports the claim about ORQA; REALM is cited separately. (LLM, conf=0.90)

  cites [31] [31] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6086-6096, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1612. URL https://www.aclweb.org/ anthology/P19-1612 . - source
• ?Not checked — no full text of the sourcecites [8] [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://www.aclweb.org/anthology/N19-1423 . - source

In the paper

“FEVER [56] requires classifying whether a natural language claim is supported or refuted by Wikipedia, or whether there is not enough information to decide.”

What the sources say

• Supported

  The Claim Labeling passage describes classifying whether Wikipedia supports or refutes a claim, or whether there is not enough information to decide. The Introduction likewise identifies the three FEVER labels as SUPPORTED, REFUTED, and NOTENOUGHINFO. (LLM, conf=0.95)

  cites [56] [56] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1074. URL https://www.aclweb.org/anthology/N18-1074 . - source

In the paper

“It has obtained state-of-the-art results on a diverse set of generation tasks and outperforms comparably-sized T5 models [32].”

What the sources say

• Supported

  Section 5.3 reports that BART outperforms previous work on the CNN/DailyMail and XSum generation tasks; the paper also reports gains on other generation tasks. Section 7 (Related Work) explicitly says BART achieves higher performance than T5 at similar model sizes, particularly on summarization. Page numbers are not available in our copy. (LLM, conf=0.93)

  cites [32] [32] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019. URL https://arxiv.org/abs/1910.13461 . - source

In the paper

“Open-domain question answering (QA) is an important real-world application and common testbed for knowledge-intensive tasks [20].”

What the sources say

• Supported

  The Background (p. 2) identifies Open-QA as a knowledge-intensive task used to measure a model’s ability to incorporate world knowledge. The Introduction (p. 2) evaluates it on three popular benchmarks, while the Open-QA Benchmarks section (p. 6) says its questions reflect realistic information-seeking needs. (LLM, conf=0.90)

  cites [20] [20] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. ArXiv , abs/2002.08909, 2020. URL https: //arxiv.org/abs/2002.08909 . - source

In the paper

“Q-BLEU is a variant of BLEU with a higher weight for matching entities and has higher correlation with human judgment for question generation than standard metrics.”

What the sources say

• Supported

  Section 5 describes Q-BLEU as BLEU combined with an answerability component that explicitly weights matching named entities; Section 5.1 shows that the weight varies by dataset. Section 5.2 reports that the modified metrics correlate more strongly with human judgments than the existing metrics tested. (LLM, conf=0.90)

  cites [42] [42] Preksha Nema and Mitesh M. Khapra. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3950-3959, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1429. URL https://www.aclweb.org/ anthology/D18-1429 . - source

In the paper

“Hybrid models that combine parametric memory with non-parametric (i.e., retrieval-based) memories [20, 26, 48] can address some of these issues because knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.”

What the sources say

• Supported

  The Introduction (p. 1) describes REALM as combining language-model pre-training with retrieval from a textual knowledge corpus, making the role of retrieved knowledge explicit. Appendix C (p. 11) shows that changing the corpus can update predictions, including when a new document is added; the Discussion (p. 8) says document-grounded retrieval provides interpretable provenance. The paper notes that some predictions may still rely on knowledge retained in model parameters, which is consistent with the claim's qualified wording, “can.” (LLM, conf=0.91)

  cites [20] [20] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. ArXiv , abs/2002.08909, 2020. URL https: //arxiv.org/abs/2002.08909 . - source
• Partly supported

  The Introduction describes a retrieval-based QA system that selects passages for a reader, and Related Work says DPR can be combined with generative models. These passages support the hybrid-model example, but our copy does not state that knowledge can be directly revised or expanded, or that accessed knowledge can be inspected and interpreted. (LLM, conf=0.82)

  cites [26] [26] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 , 2020. URL https://arxiv.org/abs/2004.04906 . - sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [48] [48] Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. In Automated Knowledge Base Construction , 2020. URL https://openreview.net/forum? id=025X0zPfn . -

In the paper

“It is worth noting that RAG's retriever is initialized using DPR's retriever, which uses retrieval supervision on Natural Questions and TriviaQA.”

What the sources say

• Partly supported

  Section 5 (Experiments: Passage Retrieval) says DPR’s retriever was trained on Natural Questions and TriviaQA, supporting the retrieval-supervision portion. Section 7 (Related Work) says DPR has been combined with generation models, but our copy does not say that RAG’s retriever was initialized from DPR. (LLM, conf=0.86)

  cites [26] [26] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 , 2020. URL https://arxiv.org/abs/2004.04906 . - sourceExperimental — accuracy not yet measured

In the paper

“BART [32] and T5 [51, 52] propose a single, pre-trained encoder-decoder model that leverages bi-directional attention to achieve stronger performance on discriminative and generative tasks.”

What the sources say

• Partly supported

  The Model section describes BART as a pre-trained sequence-to-sequence model with a bidirectional encoder and a left-to-right autoregressive decoder. The Introduction and §5.3 report improved generation results, but §5.2 says BART performs similarly to RoBERTa on discriminative tasks, not more strongly. This verdict concerns BART [32]; T5 has separate citations in the citing sentence. (LLM, conf=0.88)

  cites [32] [32] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019. URL https://arxiv.org/abs/1910.13461 . - sourceExperimental — accuracy not yet measured
• Supported

  For T5, the cited paper describes an encoder-decoder Transformer whose encoder uses a fully visible attention mask (Model Structures, p. 15). It reports good results on both generative and classification tasks (Model, p. 11), and finds that its denoising encoder-decoder variant performed best among the architectures compared (Results, p. 19). The separate citation to BART is outside this verdict. (LLM, conf=0.88)

  cites [51] [51] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints , 2019. URL https://arxiv.org/abs/1910.10683 . - source
• Partly supported

  The Background describes T5 as a pre-trained encoder-decoder Transformer and says its pre-training included classification, summarization, and other tasks. It also reports strong reading-comprehension results. Our copy does not attribute those results to bidirectional attention or establish stronger performance across both discriminative and generative tasks; this verdict concerns the T5 portion of the sentence, not BART. (LLM, conf=0.84)

  cites [52] [52] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv e-prints , 2020. URL https://arxiv.org/abs/ 2002.08910 . - sourceExperimental — accuracy not yet measured

In the paper

“There is significant work on learning to retrieve documents in information retrieval, more recently with pre-trained, neural language models [44, 26] similar to ours.”

What the sources say

• Supported

  The Abstract (p. 1) identifies the work as a BERT-based query-passage re-ranking system evaluated on passage retrieval tasks. The Introduction (p. 1) also describes substantial work on neural ranking models in information retrieval. It therefore supports the citation as an example of using a pre-trained neural language model for retrieval-related ranking. (LLM, conf=0.90)

  cites [44] [44] Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085 , 2019. URL https://arxiv.org/abs/1901.04085 . - source
• Supported

  The Introduction describes training a dense retriever using a pretrained BERT model and question–passage pairs. Section 3.1 explains that its BERT encoders retrieve passages from an index, making this paper an example of the recent neural retrieval work cited in the sentence. (LLM, conf=0.93)

  cites [26] [26] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 , 2020. URL https://arxiv.org/abs/2004.04906 . - source

In the paper

“Some work optimizes the retrieval module to aid in a specific, downstream task such as question answering, using search [46], reinforcement learning [6, 63, 62], or a latent variable approach [31, 20] as in our work.”

What the sources say

• Partly supported

  The Introduction and §2.2 describe selecting passage sentences to influence a question-answering model, so the paper supports the general point about task-directed evidence selection. But the citation is attached to “a latent variable approach,” while §2.2 describes exhaustive search and a learned predictor, not a latent-variable method. (LLM, conf=0.86)

  cites [46] [46] Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince q&a models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2402-2411, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1244. URL https://www.aclweb.org/anthology/D19-1244 . - sourceExperimental — accuracy not yet measured
• Supported

  The Introduction says the paper treats answer-sentence selection as a latent variable and trains it jointly with answer generation for question answering. The Abstract identifies this as a coarse model that selects relevant sentences before a more expensive model produces the answer, matching the cited latent-variable approach to task-focused retrieval. (LLM, conf=0.93)

  cites [6] [6] Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 209-220, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1020. URL https://www.aclweb.org/anthology/P17-1020 . - source
• Supported

  The Introduction says the Ranker selects passages for the Reader and is trained with REINFORCE using a reward based on the Reader’s answer extraction. This supports the claim’s reinforcement-learning example: passage ranking is optimized for the downstream question-answering task. (LLM, conf=0.93)

  cites [63] [63] Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. Evidence aggregation for answer reranking in open-domain question answering. In ICLR , 2018. URL https://openreview. net/forum?id=rJl3yM-Ab . - source
• Supported

  The Introduction says the Ranker selects passages for the Reader and is trained with REINFORCE using a reward based on answer extraction. It explicitly states that this optimizes ranking for downstream answer-prediction performance, supporting the citation's reinforcement-learning example. (LLM, conf=0.93)

  cites [62] [62] Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. R 3 : Reinforced ranker-reader for open-domain question answering. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 5981-5988. AAAI Press, 2018. URL http source
• Supported

  The Introduction describes ORQA as a joint retriever–reader model fine-tuned end-to-end for open-domain question answering. Section 6 (Learning) specifies optimization of the marginal log-likelihood over answer derivations, supporting its identification as a latent-variable approach to task-driven retrieval. (LLM, conf=0.93)

  cites [31] [31] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6086-6096, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1612. URL https://www.aclweb.org/ anthology/P19-1612 . - source
• Supported

  The citation to REALM is attached to the latent-variable approach, not to search or reinforcement learning. The Introduction (p. 2) describes retrieval as a latent-variable model optimized by marginal likelihood; Training (p. 4) says the retriever is optimized during both pre-training and fine-tuning, and the Abstract (p. 1) identifies Open-QA as the fine-tuning task. (LLM, conf=0.93)

  cites [20] [20] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. ArXiv , abs/2002.08909, 2020. URL https: //arxiv.org/abs/2002.08909 . - source

In the paper

“Prior work has shown that retrieval improves performance across a variety of NLP tasks when considered in isolation.”

What the sources say

• Partly supported

  Section 5.1 (p. 7) evaluates the Document Retriever separately and finds that it outperforms Wikipedia Search across the QA datasets tested. This supports an isolated retrieval improvement, but the paper does not show improved performance across a variety of NLP tasks; its comparison is limited to finding answer-containing articles for question answering. (LLM, conf=0.85)

  cites [5] [5] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1171. URL https://www.aclweb.org/anthology/P17-1171 . - sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [29] [29] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics , 2019. URL https://tomkwiat.users.x20web.corp.google.com/papers/ natural-questions/main-1455-kwiatkowski.pdf . - source
• Partly supported

  The ablation discussion (page unavailable) reports that removing sentence selection substantially reduced accuracy for some models in an oracle evaluation; the pipeline results also report an accuracy decrease without it. This supports a performance benefit for sentence selection in FEVER’s fact-verification task, but not the broader claim that retrieval improves performance across a variety of NLP tasks. (LLM, conf=0.85)

  cites [56] [56] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1074. URL https://www.aclweb.org/anthology/N18-1074 . - sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [48] [48] Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. In Automated Knowledge Base Construction , 2020. URL https://openreview.net/forum? id=025X0zPfn . -

In the paper

“A single, pre-trained language model has been shown to achieve strong performance on various classification tasks in the GLUE benchmarks [60, 61] after fine-tuning [49, 8].”

What the sources say

• Supported

  The citation to this paper is attached to “GLUE benchmarks,” while the fine-tuning and performance claim is cited separately. The benchmark description and Table 1 identify nine NLU tasks, most of them classification tasks, so this paper supports the part attributed to it. (LLM, conf=0.91)

  cites [60] [60] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://www.aclweb.org/ anthology/W18-5446 . - source
• Supported

  Related Work (p. 3) reports strong GLUE scores for pretrained models including BERT and GPT. The Introduction (p. 2) describes GLUE’s tasks as sentence and sentence-pair classification, while Baselines (p. 8) describes fine-tuning copies of a pretrained BERT model separately for tasks. Together, these passages support the claim without implying that one fine-tuned copy was used for every task. (LLM, conf=0.88)

  cites [61] [61] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A Stickier Benchmark for GeneralPurpose Language Understanding Systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d\textquotesingle Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 3261-3275. Curran Associates, Inc., 2019. URL https:// arxiv.org/abs/1905.00537 . - source
• Not checked yetcites [49] [49] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training, 2018. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/ language-unsupervised/language\_understanding\_paper.pdf . -
• ?Not checked — no full text of the sourcecites [8] [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://www.aclweb.org/anthology/N19-1423 . - source

In the paper

“On all four open-domain QA tasks, RAG sets a new state of the art (only on the T5-comparable split for TQA).”

In the paper

“Evaluators indicated that BART was more factual than RAG in only 7.1% of cases, while RAG was more factual in 42.7% of cases, and both RAG and BART were factual in a further 17% of cases”

In the paper

“Differentiable retrieval improves results on all other tasks, especially for Open-Domain QA, where it is crucial.”

In the paper

“We find that doing Maximum Inner Product Search with FAISS is sufficiently fast on CPU”

In the paper

“as Thorough Decoding did not improve performance.”

In the paper

“RAG answers 70% correctly using the 2016 index for 2016 world leaders and 68% using the 2018 index for 2018 world leaders.”

In the paper

“we illustrated how the retrieval index can be hot-swapped to update the model without requiring any retraining.”

In the paper

“Figure 3 (right) shows that retrieving more documents leads to higher Rouge-L for RAG-Token at the expense of Bleu-1, but the effect is less pronounced for RAG-Sequence.”

In the paper

“For RAG, we also make use of multiple annotation examples for Natural Questions and WebQuestions by training the model with each ( q, a ) pair separately, leading to a small increase in accuracy.”

In the paper

“Table 2 shows that RAG-Token performs better than RAG-Sequence on Jeopardy question generation, with both models outperforming BART on Q-BLEU-1.”

In the paper

“We find that the top retrieved document is from a gold article in 71% of cases, and a gold article is present in the top 10 retrieved articles in 90% of cases.”

In the paper

“For FEVER, BM25 performs best, perhaps since FEVER claims are heavily entity-centric and thus well-suited for word overlap-based retrieval.”

In the paper

“We use greedy decoding for QA as we did not find beam search improved results.”

In the paper

“Table 5 shows that RAG-Sequence's generations are more diverse than RAG-Token's, and both are significantly more diverse than BART without needing any diversity-promoting decoding.”

In the paper

“Figure 3 (left) shows that retrieving more documents at test time monotonically improves Open-domain QA results for RAG-Sequence, but performance peaks for RAG-Token at 10 retrieved documents.”

In the paper

“Accuracy with mismatched indices is low (12% with the 2018 index and 2016 leaders, 4% with the 2016 index and 2018 leaders).”

In the paper

“Qualitatively, we find that RAG models hallucinate less and generate factually correct text more often than BART.”

In the paper

“We also compress the document index using FAISS's compression tools, reducing the CPU memory requirement to 36GB.”

In the paper

“Crucially, by using pre-trained access mechanisms, the ability to access knowledge is present without additional training.”

In the paper

“To avoid any biases for screen position, which model corresponded to sentence A and sentence B was randomly selected for each example.”

In the paper

“This example shows how parametric and non-parametric memories work together -the non-parametric component helps to guide the generation, drawing out specific knowledge stored in the parametric memory.”

In the paper

“Documents with clues about the answer but do not contain the answer verbatim can still contribute towards a correct answer being generated, which is not possible with standard extractive approaches, leading to more effective marginalization over documents.”

In the paper

“Intriguingly, after the first token of each book is generated, the document posterior flattens. This observation suggests that the generator can complete the titles without depending on specific documents.”

In the paper

“We included some gold sentences in order to assess the accuracy of the annotators.”

In the paper

“Our work unifies previous successes in incorporating retrieval into individual tasks, showing that a single retrieval-based architecture is capable of achieving strong performance across several tasks.”

In the paper

“Two annotators did not perform well on these examples and their annotations were removed from the results.”

In the paper

“These successes leverage different retrieval-based architectures and optimization techniques to achieve strong performance on a single task, while we show that a single retrieval-based architecture can be fine-tuned for strong performance on a variety of tasks.”

In the paper

“Jeopardy questions often contain two separate pieces of information, and RAG-Token may perform best because it can generate responses that combine content from several documents.”

In the paper

“which we attribute to the official Wiki test set questions being simpler to answer from Wikipedia.”

In the paper

“GPT-2 [50] later showed that a single, left-to-right, pre-trained language model could achieve strong performance across both discriminative and generative tasks.”

What the sources say

• Not checked yetcites [50] [50] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language\_ models\_are\_unsupervised\_multitask\_learners.pdf . -

In the paper

“and offers more control and interpretability.”

In the paper

“An advantage of non-parametric memory models like RAG is that knowledge can be easily updated at test time. Parametric-only models like T5 or BART need further training to update their behavior as the world changes.”

In the paper

“the fact that it is more strongly grounded in real factual knowledge (in this case Wikipedia) makes it 'hallucinate' less with generations that are more factual”

In the paper

“Prior work on general-purpose architectures for NLP tasks has shown great success without the use of retrieval.”