CoChat

Self-Knowledge Guided Retrieval Augmentation for Large Language Models

Open access
Wang, Yile, Li, Peng, Sun, Maosong, Liu, Yang · 2023 · Findings of the Association for Computational Linguistics: EMNLP 2023
Cited by 61

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) have shown superior performance without task-specific finetuning.Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs.As complementary, retrieval-based methods can offer nonparametric world knowledge and improve the performance on tasks such as question answering.However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally.To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called "self-knowledge") and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions.We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either In-structGPT or ChatGPT.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.691

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 50 references

Cited by

See all 61

Metadata

Authors
Wang, Y., Li, P., Sun, M., & Liu, Y.
Venue
Findings of the Association for Computational Linguistics: EMNLP 2023
Published
2023
Pages
10303-10315
OpenAlex
W4389518895
Cited by
61
References
50
FWCI
11.53
Type
conference paper
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Self-Knowledge Guided Retrieval Augmentation for Large Language Models. https://app.cochat.ai/papers/10.18653/v1/2023.findings-emnlp.691
The paper
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023). Self-Knowledge Guided Retrieval Augmentation for Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.691

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.