CoChat
Sign Up
PreprintCited by 2Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

AnyEdit: Edit Any Knowledge Encoded in Language Models

Open access
Jiang, Houcheng, Fang, Junfeng, Zhang, Ningyu, Ma, Guojun, Wan, Mingyang, Wang, Xiang, He, Xiangnan, Chua, Tat-seng·2025·arXiv
DOI 10.48550/arxiv.2502.05628arXiv 2502.05628
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token's hidden state, a limitation we term "efficacy barrier". To solve this, we propose AnyEdit, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Information, showing its ability to update any knowledge within LLMs. Empirically, it outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current editing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.05628

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 40

  1. Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. · arXiv
  2. Bi, B., Liu, S., Mei, L., Wang, Y., Ji, P., and Cheng, X. Decoding by contrasting knowledge: Enhancing llms' confidence on edited facts. CoRR, abs/2405.11613, 2024. · arXiv
  3. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS, 2020.
  4. Cao, N. D., Aziz, W., and Titov, I. Editing factual knowledge in language models. In EMNLP (1), pp. 6491-6506. Association for Computational Linguistics, 2021.
  5. Chen, C., Huang, B., Li, Z., Chen, Z., Lai, S., Xu, X., Gu, J., Gu, J., Yao, H., Xiao, C., Yan, X., Wang, W. Y., Torr, P., Song, D., and Shu, K. Can editing llms inject harm? CoRR, abs/2407.20224, 2024. · arXiv
  6. Deng, J., Wei, Z., Pang, L., Ding, H., Shen, H., and Cheng, X. Everything is editable: Extend knowledge editing to unstructured data in large language models. In ICLR. OpenReview.net, 2025.
  7. Dong, Q., Dai, D., Song, Y., Xu, J., Sui, Z., and Li, L. Cali- brating factual knowledge in pretrained language models. In EMNLP (Findings), pp. 5937-5947. Association for Computational Linguistics, 2022.
  8. Fang, J., Jiang, H., Wang, K., Ma, Y., Shi, J., Wang, X., He, X., and Chua, T. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR. Open- Review.net, 2025.
  9. Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In EMNLP (1), pp. 5484-5495. Association for Computational Lin- guistics, 2021.
  10. Gu, J., Xu, H., Ma, J., Lu, P., Ling, Z., Chang, K., and Peng, N. Model editing harms general abilities of large language models: Regularization to the rescue. In EMNLP, pp. 16801-16819. Association for Computational Linguistics, 2024.
  11. Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurIPS, 2023.
  12. Huang, B., Chen, C., Xu, X., Payani, A., and Shu, K. Can knowledge editing really correct hallucinations? In ICLR. OpenReview.net, 2025.
  13. Huang, X., Wang, Y., Zhao, J., and Liu, K. Commonsense knowledge editing based on free-text in llms. In EMNLP, pp. 14870-14880. Association for Computational Lin- guistics, 2024.
  14. Huang, Z., Shen, Y., Zhang, X., Zhou, J., Rong, W., and Xiong, Z. Transformer-patcher: One mistake worth one neuron. In ICLR. OpenReview.net, 2023.
  15. Jiang, H., Fang, J., Zhang, T., Zhang, A., Wang, R., Liang, T., and Wang, X. Neuron-level sequential editing for large language models. CoRR, abs/2410.04045, 2024. · arXiv
  16. Lang, S. Introduction to linear algebra. Springer Science & Business Media, 2012.
  17. Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. CAMEL: communicative agents for "mind" exploration of large scale language model society. CoRR, abs/2303.17760, 2023. · arXiv
  18. Li, Z., Jiang, H., Chen, H., Bi, B., Zhou, Z., Sun, F., Fang, J., and Wang, X. Reinforced lifelong editing for language models. CoRR, abs/2502.05759, 2025. · arXiv
  19. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
  20. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173, 2024a.
  21. Liu, Y., He, X., Xiong, M., Fu, J., Deng, S., and Hooi, B. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832, 2024b. · arXiv
  22. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. In NeurIPS, 2022.
  23. Meng, K., Sharma, A. S., Andonian, A. J., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. In ICLR. OpenReview.net, 2023.
  24. Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In ICLR. OpenRe- view.net, 2022a.
  25. Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. Memory-based model editing at scale. In ICML, vol- ume 162 of Proceedings of Machine Learning Research, pp. 15817-15831.
  26. PMLR, 2022b.
  27. Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math. CoRR, abs/2402.14830, 2024. · arXiv
  28. Papineni, K., Roukos, S., Ward, T., and Zhu, W. Bleu: a method for automatic evaluation of machine translation. In ACL, pp. 311-318. ACL, 2002.
  29. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  30. Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., and Chen, H. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405.14768, 2024. · arXiv
  31. Wu, X., Pan, L., Wang, W. Y., and Luu, A. T. AKEW: assessing knowledge editing in the wild. In EMNLP, pp. 15118-15133. Association for Computational Linguistics, 2024.
  32. Xie, J., Zhang, K., Chen, J., Lou, R., and Su, Y. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge clashes. CoRR, abs/2305.13300, 2023. · arXiv
  33. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. · arXiv
  34. Zhang, N., Tian, B., Cheng, S., Liang, X., Hu, Y., Xue, K., Gou, Y., Chen, X., and Chen, H. Instructedit: Instruction- based knowledge editing for large language models. In IJCAI, pp. 6633-6641. ijcai.org, 2024.
  35. Zhang, Q., Chen, H., Dong, J., Chen, S., Huang, F., and Huang, X. Structure-guided large language models for text-to-SQL generation. In Forty-second International Conference on Machine Learning, 2025.
  36. Zhang, T., Fang, J., Jiang, H., Bi, B., Wang, X., and He, X. Explainable and efficient editing for large language models. In THE WEB CONFERENCE 2025.
  37. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. · arXiv
  38. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models, 2024. URL https://arxiv. org/abs/2303.18223.
  39. Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., and Chang, B. Can we edit factual knowledge by in-context learning? In EMNLP, pp. 4862-4876. Association for Computational Linguistics, 2023.
  40. Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F. X., and Kumar, S. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020. Method Lexical Similarity Semantic Similarity Sub Questions BLEU ROUGE-1 ROUGE-2 ROUGE-L BERT Score ROUGE-L Based on Llama3-8B-Instruct UnKE 93.56 / 78.09 93.61 / 79.26 91.42 / 71.73 93.33 / 78.42 98.34 / 93.38 37.87 UnKE+ 99.67 / 84.60 99.69 / 86.31 99.57 / 81.18 99.68 / 85.75 99.86 / 94.70 41.45 MEMIT 25.57 / 22.88 32.67 / 30.75 14.51 / 12.37 30.49 / 28.65 76.21 / 74.25 22.56 MEMIT+ 88.88 / 81.38 93.26 / 86.53 90.32 / 80.61 92.96 / 85.91 97.76 / 95.60 41.67 AlphaEdit 21.29 / 20.24 28.62 / 27.99 11.36 / 10.24 26.59 / 25.92 73.92 / 72.96 20.71 AlphaEdit+ 75.02 / 66.35 81.70 / 73.47 74.35 / 62.74 80.92 / 72.22 94.19 / 91.51 40.56 Based on Qwen2.5-7B-Instruct UnKE 91.92 / 70.61 91.41 / 68.47 87.75 / 56.34 91.01 / 67.00 96.97 / 89.17 38.12 UnKE+ 98.52 / 82.48 98.85 / 83.36 98.43 / 77.03 98.82 / 82.60 99.35 / 94.81 42.24 MEMIT 45.07 / 40.81 40.73 / 36.75 19.59 / 15.87 38.04 / 34.07 78.03 / 76.50 24.75 MEMIT+ 91.31 / 77.23 95.10 / 80.88 92.93 / 72.50 94.89 / 79.98 98.05 / 93.56 42.38 AlphaEdit 49.71 / 45.21 45.42 / 41.06 24.63 / 19.85 42.77 / 38.26 80.48 / 78.38 25.37 AlphaEdit+ 97.77 / 83.09 98.20 / 84.18 97.40 / 77.38 98.14 / 83.40 99.08 / 94.51 41.58 C.3. Supplementary Experimental Results on RQ4 The experimental results of relationship between AnyEdit's editing performance and chunk size in long-form diverse- formatted knowledge are presented in Figure 6. Based on these results, we draw the following observation:. • Obs 7: The editing performance of AnyEdit is influenced by chunk size. As the chunk size increases beyond a certain threshold, the editing performance of AnyEdit declines. Specifically, when the chunk size is smaller, each iteration of editing becomes more manageable, leading to improved overall performance. However, this improvement comes at the cost of increased editing time due to the larger number of iterations required for longer texts. Conversely, when the chunk size is larger, it becomes challenging to achieve effective edits within a single iteration, resulting in degraded performance. Based on this trade-off, we recommend using a balanced chunk size of 40 for most editing scenarios. Table · arXiv

Cited by

Sorted by citations

Authors
Jiang, H., Fang, J., Zhang, N., Ma, G., Wan, M., Wang, X., He, X., & Chua, T.-S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.05628
arXiv
2502.05628
OpenAlex
W4407385068
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science and Technology Major Project
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
knowledge updating, code editing, poetry editing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review