CiteME: Can Language Models Accurately Cite Scientific Claims?
Open accessAbstract
Thousands of new scientific papers are published each month. Such information overload complicates researcher efforts to stay current with the state-of-the-art as well as to verify and correctly attribute claims. We pose the following research question: Given a text excerpt referencing a paper, could an LM act as a research assistant to correctly identify the referenced paper? We advance efforts to answer this question by building a benchmark that evaluates the abilities of LMs in citation attribution. Our benchmark, CiteME, consists of text excerpts from recent machine learning papers, each referencing a single other paper. CiteME use reveals a large gap between frontier LMs and human performance, with LMs achieving only 4.2-18.5% accuracy and humans 69.7%. We close this gap by introducing CiteAgent, an autonomous system built on the GPT-4o LM that can also search and read papers, which achieves an accuracy of 35.3\% on CiteME. Overall, CiteME serves as a challenging testbed for open-ended claim attribution, driving the research community towards a future where any claim made by an LM can be automatically verified and discarded if found to be incorrect.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CiteME is limited in scope, restricting its application to other fields.
“While CiteME addresses a crucial aspect of literature review by ensuring accurate citation, it is limited in scope, restricting its application to other fields.”
From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Press et al., 2024
The paper describes CiteME as a benchmark of recent machine-learning publications, supporting the claim that its evaluated scope is limited. But the Conclusion says methods that succeed on CiteME may also be useful for finding sources for generic claims; it does not establish that the benchmark’s scope restricts application to other fields. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCitation attribution benchmarks such as CiteME and CiteGuard test whether a model attributes a scientific claim to the correct source paper, a matching task over a fixed corpus.
“Citation attribution benchmarks such as CiteME [Press et al., 2024] and CiteGuard [Choi et al., 2026] test whether a model attributes a scientific claim to the correct source paper, a matching task over a fixed corpus.”
From Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution (2026) · cites this paper as Press et al., 2024
The Abstract describes CiteME as testing whether a model can identify the paper referenced by a scientific claim. But the Introduction distinguishes CiteME from benchmarks that retrieve from a small, pre-selected set of papers, and the contributions describe its attribution task as open-ended rather than limited to a fixed corpus. This verdict addresses CiteME, not the separately cited CiteGuard. (LLM, conf=0.88)
Experimental — accuracy not yet measuredCiteME evaluates citation matching in academic contexts, revealing that LLMs achieve only 4-18% accuracy on identifying correct papers to cite.
“CiteME (Press et al., 2024) evaluates citation matching in academic contexts, revealing that LLMs achieve only 4-18% accuracy on identifying correct papers to cite.”
From Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents (2026) · cites this paper as Press et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Pepa Atanasova. Generating fact checking explanations. In Accountable and Explainable Methods for Complex Reasoning over Text, pages 83-103. Springer, 2024.
- Yoshua Bengio. Deep learning of representations: Looking forward. In International conference on statistical language and speech processing, pages 1-37. Springer, 2013.
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206-2240. PMLR, 2022.
- Thang Bui, Daniel Hernández-Lobato, Jose Hernandez-Lobato, Yingzhen Li, and Richard Turner. Deep gaussian processes for regression using approximate expectation propagation. In International conference on machine learning, pages 1472-1481. PMLR, 2016.
- David R Burt, Carl Edward Rasmussen, and Mark Van Der Wilk. Convergence of sparse variational inference in gaussian processes regression. Journal of Machine Learning Research, 21(131):1-63, 2020.
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
- David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20(2):215-232, 1958.
- Nianlong Gu, Yingqiang Gao, and Richard HR Hahnloser. Local citation recommendation with hierarchical- attention text encoder and scibert-based reranking. In European conference on information retrieval, pages 274-288. Springer, 2022.
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
- Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129-137, 1982.
- Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. 2024. URL https://api.semanticscholar.org/CorpusID: 269757000.
- John F. Wu, Alina Hyk, Kiera McCormick, Christine Ye, Simone Astarita, Elina Baral, Jo Ciuca, Jesse Cranney, Anjalie Field, Kartheik G. Iyer, Philipp Koehn, Jenn Kotler, Sandor J. Kruk, Michelle Ntampaka, Charlie O'Neill, Josh Peek, Sanjib Sharma, and Mikaeel Yunus. Designing an evaluation framework for large language models in astronomy research. 2024. URL https://api.semanticscholar.org/ CorpusID:270199896.
Cited by
Sorted by citations
(Generative) AI in Financial Economics
Mo, H., & Ouyang, S. (2025) · Journal of Chinese Economic and Business Studies · doi:10.1080/14765284.2025.2569006 · Cited by 28 · see citing works
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.840 · Cited by 23 · see citing works
Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions
Gridach, M., Nanavati, J., Abidine, K. Z. E., Mendes, L., & Mack, C. (2025) · arXiv · doi:10.48550/arxiv.2503.08979 · Cited by 18 · see citing works
“Prominent frameworks, including LitSearch (Ajith et al., 2024), ResearchArena (Kang & Xiong, 2024), SciLitLLM (Li et al., 2024c), CiteME (Press et al., 2024), ResearchAgent (Baek et al., 2024) and Agent Laboratory (Schmidgall et al., 2025), have made strides in automating general research…”
InfluentialLiterature Meets Data: A Synergistic Approach to Hypothesis Generation
Liu, H., Zhou, Y., Li, M., Yuan, C., & Tan, C. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.12 · Cited by 13 · see citing works
Zhong, Y., Leng, Y., Gu, Z., Guo, S., Li, P., Das, S., Liu, Y., Wan, J., & Liu, Y. (2025) · Journal of Materials Chemistry A · doi:10.1039/d5ta05224f · Cited by 11 · see citing works
IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery
Garikaparthi, A., Patwardhan, M. S., Vig, L., & Cohan, A. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-demo.57 · Cited by 6 · see citing works
XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration
Chen, N., HuiKai, A. L., Wu, J., Hou, J., Zhang, Z., Wang, Q., Wang, X., & He, B. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.47 · Cited by 6 · see citing works
Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11363 · Cited by 4 · see citing works
“…machine learning experiments (Huang et al., 2023), research programming (Tian et al., 2024), scientific discovery (Majumder et al., 2024), performing scientific reasoning and citation tasks (Press et al., 2024; Xu et al., 2024), and solving real-world programming problems (Zhang et al., 2024).”
Authorship Drift: How Self-Efficacy and Trust Evolve During LLM-Assisted Writing
Park, Y. S., Arvi, N. A. P., Kim, S., & Kim, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790276 · Cited by 4 · see citing works
Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
Onweller, H., Lumer, E., Huber, A., Ramchandani, P., Subbiah, V. K., & Feld, C. (2026) · arXiv · doi:10.48550/arxiv.2605.06635 · Cited by 2 · see citing works
“CiteGuard (Choi et al., 2025) reframes citation evaluation as attribution alignment, achieving 68% accuracy on CiteME through retrieval-aware verification.”
CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validation
Choi, Y. M., Guo, X., Fung, Y. R., & Wang, Q. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.282 · Cited by 1 · see citing works
Leung, E., Lumer, E., Feld, C., Huber, A., Subbiah, V. K., & Paul, K. (2026) · arXiv · doi:10.48550/arxiv.2607.08700
“Citation attribution benchmarks such as CiteME [Press et al., 2024] and CiteGuard [Choi et al., 2026] test whether a model attributes a scientific claim to the correct source paper, a matching task over a fixed corpus.”
MIRAGE: A Metrics lIbrary for Rating hAllucinations in Generated tExt
Vendeville, B., Магдановна, Е. Л., Loor, P. D., & Kamps, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761644
- Authors
- Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., & Bethge, M.
- ORCID
- Bethge, Matthias
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 7847-7877
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2407.12861
- OpenAlex
- W4415798685
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 1 · 2025: 3
- FWCI
- 1.15
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
- Good Ventures Foundation
- Deutsche Forschungsgemeinschaft (206/4-1, 276693517, 390727645, SFB1233)
- International Max Planck Research School for Advanced Methods in Process and Systems Engineering
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Keywords
- citation generation, language models, scientific claims, citation accuracy, large language models, information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
