CoChat
Sign Up
Cited by 42

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Evaluating the Ripple Effects of Knowledge Editing in Language Models

Cohen, Roi, Biran, Eden, Yoran, Ori, Globerson, Amir, Geva, Mor·2024·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00644
Open this paper in CoChat — free account

Abstract

Modern language models capture a large body of factual knowledge. However, some facts can be incorrectly induced or become obsolete over time, resulting in factually incorrect generations. This has led to the development of various editing methods that allow updating facts encoded by the model. Evaluation of these methods has primarily focused on testing whether an individual fact has been successfully injected, and if similar predictions for other subjects have not changed. Here we argue that such evaluation is limited, since injecting one fact (e.g., “Jack Depp is the son of Johnny Depp”) introduces a “ripple effect” in the form of additional facts that the model needs to update (e.g., “Jack Depp is the sibling of Lily-Rose Depp”). To address this, we propose novel evaluation criteria that consider the implications of an edit on related facts. Using these criteria, we then construct RippleEdits, a diagnostic benchmark of 5K factual edits, capturing various types of ripple effects. We evaluate prominent editing methods on RippleEdits, showing that they fail to introduce consistent changes in the model’s knowledge. In addition, we find that a simple in-context editing baseline obtains the best scores on our benchmark, suggesting a promising research direction for model editing.1

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00644

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Illustration of the evaluation scope of RIPPLEEDITS, compared to existing knowledge editing benchmarks. For a given factual edit, we consider the ''ripple effect'' of the edit on the model's knowledge.
Figure 1 Illustration of the evaluation scope of RIPPLEEDITS, compared to existing knowledge editing benchmarks. For a given factual edit, we consider the ''ripple effect'' of the edit on the model's knowledge.CC-BY · From the PDF
Figure 2 An illustration of our evaluation criteria, for an edit that simulates adding a sibling to the subject entity Prince , shown at the top of each graph with a bold arrow and an edit sign over the Sibling relation. For each criterion, the tested subject and target object are circles with dashed blue line and solid green line , respectively, and other nodes in dotted orange line . For Logical Generalization (A), the additional fact that needs to be inserted to the knowledge graph (KG) is presented with an edit sign next to the relation. We show the same node in different colors for completeness, as the tested subject is also the object in the edit that needs to be inserted. For Compositionality I, II (B, C), the model needs to hop over the edit to arrive at the target. In Subject Aliasing (D) we verify the edit also propagates to paraphrases of the input. In Preservation (E), we verify that other targets of the edited subject-relation are preserved. In Relation Specificity , we verify other relations for the subject are not modified.
Figure 2 An illustration of our evaluation criteria, for an edit that simulates adding a sibling to the subject entity Prince , shown at the top of each graph with a bold arrow and an edit sign over the Sibling relation. For each criterion, the tested subject and target object are circles with dashed blue line and solid green line , respectively, and other nodes in dotted orange line . For Logical Generalization (A), the additional fact that needs to be inserted to the knowledge graph (KG) is presented with an edit sign next to the relation. We show the same node in different colors for completeness, as the tested subject is also the object in the edit that needs to be inserted. For Compositionality I, II (B, C), the model needs to hop over the edit to arrive at the target. In Subject Aliasing (D) we verify the edit also propagates to paraphrases of the input. In Preservation (E), we verify that other targets of the edited subject-relation are preserved. In Relation Specificity , we verify other relations for the subject are not modified.CC-BY · From the PDF
Figure 3 Illustration of our data generation process. We start by sampling a fact from a KG (A), here ( Bill Gates , Spouse , Melinda Gates ) . Then, we generate the target triplet for the edit (B), in this case, choosing an object ( Ricciarda Cybo Malaspina ) that shares the same type as the original object. Next, we generate test queries (C) by sampling new triplets from the KG that should be retained or modified post-editing. Last, we utilize pre-defined templates to translate the KG triplets to natural language phrases (D).
Figure 3 Illustration of our data generation process. We start by sampling a fact from a KG (A), here ( Bill Gates , Spouse , Melinda Gates ) . Then, we generate the target triplet for the edit (B), in this case, choosing an object ( Ricciarda Cybo Malaspina ) that shares the same type as the original object. Next, we generate test queries (C) by sampling new triplets from the KG that should be retained or modified post-editing. Last, we utilize pre-defined templates to translate the KG triplets to natural language phrases (D).CC-BY · From the PDF
Figure 4 Most frequent relations and their frequency, in each subset of RIPPLEEDITS.
Figure 4 Most frequent relations and their frequency, in each subset of RIPPLEEDITS.CC-BY · From the PDF
Figure 5 An example modification edit from our ICE baseline. The color code of the KG is similar to that described in Figure 2. We prepend the prefix '' Imagine that '' to the input prompt, as counterfactuals can contradict knowledge embedded in a model's parameters.
Figure 5 An example modification edit from our ICE baseline. The color code of the KG is similar to that described in Figure 2. We prepend the prefix '' Imagine that '' to the input prompt, as counterfactuals can contradict knowledge embedded in a model's parameters.CC-BY · From the PDF
Figure 6 Accuracy averaged over evaluation criteria of ROME, as a function of the model's number of parameters, for the following models: GPT2-M, GPT2-L, GPT2-XL, GPT-J, LLAMA, and GPT-NEO.
Figure 6 Accuracy averaged over evaluation criteria of ROME, as a function of the model's number of parameters, for the following models: GPT2-M, GPT2-L, GPT2-XL, GPT-J, LLAMA, and GPT-NEO.CC-BY · From the PDF
Figure 7 The average accuracy of GPT-2 on different evaluation criteria in RIPPLEEDITS. Results are averaged over editing methods (ROME, MEMIT, and MEND); error bars indicate standard deviation.
Figure 7 The average accuracy of GPT-2 on different evaluation criteria in RIPPLEEDITS. Results are averaged over editing methods (ROME, MEMIT, and MEND); error bars indicate standard deviation.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 26

  1. Black, 2022, GPT-NeoX-20B: An open-source autoregressive language model, 95, 10.18653/v1/2022.bigscience-1.9 · DOI
  2. Cohen, 2023, Crawling the internal knowledge-base of language models, 1856, 10.18653/v1/2023.findings-eacl.139 · DOI
  3. Dhingra, 2022, Time-aware language models as temporal knowledge bases, Transactions of the Association for Computational Linguistics, 10, 257, 10.1162/tacl_a_00459 · DOI · Paper page
  4. Genin, 2022, Formal representations of belief
  5. Geva, 2021, Transformer feed-forward layers are key-value memories, 5484, 10.18653/v1/2021.emnlp-main.446 · DOI · Paper page
  6. Hase, 2023, Methods for measuring, updating, and visualizing factual beliefs in language models, 2714, 10.18653/v1/2023.eacl-main.199 · DOI
  7. Heinzerling, 2021, Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries, 1772, 10.18653/v1/2021.eacl-main.153 · DOI · Paper page
  8. Hernandez, 2023, Measuring and manipulating knowledge representations in language models, ArXiv preprint
  9. Hoelscher-Obermaier, 2023, Detecting edit failures in large language models: An improved specificity benchmark, 11548, 10.18653/v1/2023.findings-acl.733 · DOI
  10. Kassner, 2021, BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief, 8849, 10.18653/v1/2021.emnlp-main.697 · DOI
  11. Lewis, 2020, Retrieval- augmented generation for knowledge-intensive NLP tasks
  12. Meng, 2023, Mass-editing memory in a transformer
  13. Mitchell, 2022, Fast model editing at scale
  14. Onoe, 2023, Can LMs learn new entities from descriptions? Challenges in propagating injected knowledge, 5469, 10.18653/v1/2023.acl-long.300 · DOI
  15. Petroni, 2019, Language models as knowledge bases?, 2463, 10.18653/v1/D19-1250 · DOI · Paper page
  16. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  17. Roberts, 2020, How much knowledge can you pack into the parameters of a language model?, 5418, 10.18653/v1/2020.emnlp-main.437 · DOI · Paper page
  18. Shin, 2020, AutoPrompt: Eliciting knowledge from language models with automatically generated prompts, 4222, 10.18653/v1/2020.emnlp-main.346 · DOI · Paper page
  19. Si, 2023, Prompting GPT-3 to be reliable
  20. Wang, 2021, K-Adapter: Infusing knowledge into pre-trained models with adapters, 1405, 10.18653/v1/2021.findings-acl.121 · DOI
  21. Yao, 2022, Kformer: Knowledge injection in transformer feed-forward layers, 131, 10.1007/978-3-031-17120-8_11 · DOI
  22. Yao, 2023, Editing large language models: Problems, methods, and opportunities, 10.18653/v1/2023.emnlp-main.632 · DOI
  23. Zhang, 2021, Drop redundant, shrink irrelevant: Selective knowledge injection for language pretraining, 4007, 10.24963/ijcai.2021/552 · DOI
  24. Zhang, 2022, Greaselm: Graph reasoning enhanced language models
  25. Zhang, 2019, ERNIE: Enhanced language representation with informative entities, 1441, 10.18653/v1/P19-1139 · DOI · Paper page
  26. Zhong, 2023, Mquake: Assessing knowledge editing in language models via multi-hop questions, 10.18653/v1/2023.emnlp-main.971 · DOI

Cited by

Sorted by citations

Metadata

Authors
Cohen, R., Biran, E., Yoran, O., Globerson, A., & Geva, M.
Affiliations
  • Blavatnik School of Computer Science, Tel Aviv University, Israel. roi1@mail.tau.ac.il
  • Blavatnik School of Computer Science, Tel Aviv University, Israel. edenbiran@mail.tau.ac.il
  • Blavatnik School of Computer Science, Tel Aviv University, Israel. oriy@mail.tau.ac.il
  • Blavatnik School of Computer Science, Tel Aviv University, Israel. gamir@tauex.tau.ac.il
  • Blavatnik School of Computer Science, Tel Aviv University, Israel. morgeva@tauex.tau.ac.il
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2024
Volume
12
Pages
283-298
ISSN
2307-387X
DOI
10.1162/tacl_a_00644
Cited by
42
References
21
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor, 102 d)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review