The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks
Open accessAbstract
Tzu-Ling Lin, Wei-Chih Chen, Teng-Fang Hsiao, Hou-I Liu, Ya-Hsin Yeh, Yu-Kai Chan, Wen-Sheng Lien, Po-Yen Kuo, Philip S. Yu, Hong-Han Shuai. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their evaluations can also be influenced by manuscript-level factors, including hidden prompt injection, adversarial textual perturbations, metadata-related biases, and presentation-related variation (Ye et al., 2024; Collu et al., 2026; Lin et al., 2025; Vasu et al., 2026).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Lin et al., 2025
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hidden instructions embedded in manuscripts can inflate ratings, suppress criticism, or redirect generated reviews (Ye et al., 2024; Collu et al., 2026), while character-, word-, and sentence-level perturbations can distort review judgments (Lin et al., 2025).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Lin et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 9
- Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Sri- nath, et al. 2024. Llms assist nlp researchers: Cri- tique paper (meta-) reviewing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.
- Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial at- tack against bert using bert. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing.
- Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021a. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing.
- Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021b. Hidden killer: Invisible textual backdoor at- tacks with syntactic trigger. In Proceedings of the An- nual Meeting of the Association for Computational.
- Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. In Proceedings of the Neural Information Processing Systems Workshop on Instruction Tuning and Instruction Following.
- Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. 2018. De- sign and analysis of the nips 2016 review process. Journal of machine learning research.
- Junchao Wu, Runzhe Zhan, Derek Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia Chao. 2024. De- tectrl: Benchmarking llm-generated text detection in real-world scenarios. Advances in Neural Informa- tion Processing Systems.
- Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Ren- Jing Cui, Chengcheng Han, Qiushi Sun, et al. 2024a. Automated peer reviewing in paper sea: Standard- ization, evaluation, and analysis. In Findings of the Association for Computational Linguistics: EMNLP.
- Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2024. Promptbench: A unified library for evaluation of large language models. Journal of machine learning research.
Cited by
Sorted by citations
The more you automate, the less you see: Hidden pitfalls of autonomous AI scientists
Luo, Z., Kasirzadeh, A., & Shah, N. B. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2610214123 · Cited by 14
When AI reviews science: Can we trust the referee?
Wang, J., Liu, Y., Xu, H., Hu, K., Di, S., Ni, W., Yue, L., Zhang, M.-L., Ren, K., & Chen, L. (2026) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2026.100030 · Cited by 7
Torrielli, F., Locci, S., Rapp, A., & Di, L. (2026) · Scientometrics · doi:10.1007/s11192-026-05695-x · Cited by 6
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Li, M., Wang, C., Li, X., Zeng, X., Li, D., Shi, P., Zhou, D., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2608.08975
LLM-Assisted Reviewer Assignment via Auditable Expertise Matching
Bagheri, F., Buscaldi, D., & Recupero, D. R. (2026) · IEEE Access · doi:10.1109/access.2026.3724622
Kim, E., & Singh, V. (2026) · Publications · doi:10.3390/publications14040063
Artificial Intelligence in Research and Academic Publication
Zhang, Y., & Xie, Q. (2026) · IEEE Intelligent Systems · doi:10.1109/mis.2026.3726702
Metadata
- Authors
- Lin, T.-L., Chen, W.-C., Hsiao, T.-F., Liu, H.-I., Yeh, Y.-H., Chan, Y.-K., Lien, W.-S., Kuo, P.-Y., Yu, P. S., & Shuai, H.-H.
- Affiliations
- National Yang Ming Chiao Tung University (TW)
- University of Illinois Chicago (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2025
- Published
- 2025
- Pages
- 4819-4839
- OpenAlex
- W4416034894
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 6
- FWCI
- 5.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (2346158, 2106758)
- National Science and Technology Council (NSTC-112-2221-E-A49-094-MY3)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, automated peer review, adversarial attacks, vulnerability assessment, academic writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
