PMET: Precise Model Editing in a Transformer
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the \textsc{counterfact} and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at \url{https://github.com/xpq-tech/PMET}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Zero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1706.04115 · Cited by 482
Transformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2020) · arXiv · doi:10.48550/arxiv.2012.14913 · Cited by 353
Transformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.446 · Cited by 353
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230
Editing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180
Evaluating the Ripple Effects of Knowledge Editing in Language Models
Cohen, R., Biran, E., Yoran, O., Globerson, A., & Geva, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00644 · Cited by 134
(2023) · doi:10.18653/v1/2023.emnlp-main.632 · Cited by 115
Cited by
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
Self-citation(2023) · doi:10.18653/v1/2023.emnlp-main.632 · Cited by 115
Knowledge Editing for Large Language Models: A Survey
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., & Li, J. (2025) · ACM Computing Surveys · doi:10.1145/3698590 · Cited by 79
FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG
Dassen, M., Kotula, R., Murray, K., Yates, A., Lawrie, D., Kayi, E., Mayfield, J., & Duh, K. (2026) · arXiv · doi:10.48550/arxiv.2601.05866
Metadata
- Authors
- Li, X., Li, S., Song, S., Yang, J., Ma, J., & Yu, J.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 17
- Pages
- 18564-18572
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393147125
- Cited by
- 44
- References
- 21
- FWCI
- 11.86
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: PMET: Precise Model Editing in a Transformer. https://app.cochat.ai/papers/10.1609/aaai.v38i17.29818
- The paper
- Li, X., Li, S., Song, S., Yang, J., Ma, J., & Yu, J. (2024). PMET: Precise Model Editing in a Transformer. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v38i17.29818
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.