The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 27 of the 43 sources its claims cite were checked for support.
Synthesizing scientific literature with retrieval-augmented language models
Open accessAbstract
Scientific progress depends on the ability of researchers to synthesize the growing body of literature. Can large language models (LLMs) assist scientists in this task? Here we introduce OpenScholar, a specialized retrieval-augmented language model (LM)1 that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience and biomedicine. Despite being a smaller open model, OpenScholar-8B outperforms GPT-4o by 6.1% and PaperQA2 by 5.5% in correctness on a challenging multi-paper synthesis task from the new ScholarQABench. Although GPT-4o hallucinates citations 78–90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar’s data store, retriever and self-feedback inference loop improve off-the-shelf LMs: for instance, OpenScholar-GPT-4o improves the correctness of GPT-4o by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT-4o responses over expert-written ones 51% and 70% of the time, respectively, compared with 32% for GPT-4o. We open-source all artefacts, including our code, models, data store, datasets and a public demo. A specialized, open-source, retrieval-augmented language model is introduced for answering scientific queries and synthesizing literature, the responses of which are shown to be preferred by human evaluations over expert-written answers.
Overview
CoChat Summary
TL;DR
OpenScholar is a fully open retrieval-augmented language model that answers scientific queries by retrieving passages from 45 million open-access papers and synthesizing citation-backed responses. On the new ScholarQABench, OpenScholar-8B outperforms GPT-4o and PaperQA2 on multi-paper synthesis correctness, and OpenScholar variants are often preferred over expert-written answers in human evaluations.
Key findings
- OpenScholar-8B outperforms GPT-4o by 6.1% and PaperQA2 by 5.5% in correctness on a challenging multi-paper synthesis task from ScholarQABench; OpenScholar-GPT-4o improves GPT-4o correctness by 12%.
- GPT-4o hallucinates citations 78–90% of the time, whereas OpenScholar achieves citation accuracy on par with human experts; in Table 2, OpenScholar-8B had 0 hallucinated papers in computer science and biomedicine, versus 78.1–97.6% hallucination ratios for Llama 3.1 and GPT-4o without retrieval.
- In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT-4o responses over expert-written ones 51% and 70% of the time, respectively, compared with 32% for GPT-4o.
- On Scholar-CS rubric scores, OpenScholar-GPT-4o scored 57.7, OpenScholar-8B 51.1, human experts 54.0, PaperQA2 45.6 and GPT-4o 45.0; OpenScholar-8B also outperformed PaperQA2 and Perplexity Pro by 6% and 10%, respectively.
Methods
- OpenScholar combines a domain-specialized data store (OSDS) of 45 million scientific papers and 236 million passage embeddings, trained retrievers/rerankers, citation-aware generation, and a self-feedback iterative RAG loop with citation verification.
- ScholarQABench was built with 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience and biomedicine; it includes single-paper tasks (PubMedQA, SciFact, QASA) and multi-paper long-form tasks (Scholar-CS, Scholar-Multi, Scholar-Bio, Scholar-Neuro).
- Evaluation used automatic metrics (correctness, citation F1, LLM-judge scores) plus human rubric-based assessments; expert evaluations involved 16 PhD-level raters and 108 questions from Scholar-Multi.
Limitations
- The paper states that expert annotation is costly, so human-written evaluation sets are small (for example, 110 for computer science long-form QA; 108 expert answers), which may introduce variance and annotator-expertise bias.
- The paper states that automatic evaluation may not perfectly capture quality: Scholar-CS combines length, excerpts and rubric items with heuristic weights; citation precision/recall is sentence-level and can be overly strict; annotations reflect specific time points (July 2024 for Scholar-CS, September 2024 for Scholar-Multi); and the static public benchmark raises future contamination risks.
- The paper states that ScholarQABench primarily covers computer science, biomedicine and physics, with no social sciences or other engineering/scientific disciplines, so findings may not fully generalize; OpenScholar also does not consistently retrieve the most representative papers, may contain factual inaccuracies or unsupported information (especially the 8B model), and OpenScholar-GPT-4o relies on the proprietary GPT-4o API, making exact replication challenging.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
15 Valid · 11 Not valid · 8 Not checked · 124 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“However, passages retrieved by the bi-encoder may include unhelpful context owing to deep interactions between a query and passages, as they are encoded separately30.”
§ Synthesizing scientific literature with retrieval-augmented language models > Methods > OpenScholar
What the sources say
- Disputedcites [30] Asai, A. et al. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023 (eds Rogers, A., Boyd-Graber, J. & Okazaki, N.) 3650–3675 (ACL, 2023). sourceExperimental — accuracy not yet measured
Section 4.1 says the dual encoder encodes queries and documents independently, but describes its limitation as *limited* query–document interactions, not deep interactions. The Limitations section likewise hypothesizes that limited interactions contribute to performance degradation. The claim reverses the proposed reason for unhelpful retrieval. (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Evaluations for literature synthesis are also limited, typically focusing on narrow, single-discipline studies8,9 or simplified tasks such as multiple-choice question answering10.”
§ Synthesizing scientific literature with retrieval-augmented language models > Main
What the sources say
- Could not verifycites [8] Agarwal, S., Laradji, I.H., Charlin, L. & Pal, C. Litllm: a toolkit for scientific literature review. Preprint at https://arxiv.org/abs/2402.01788v1 (2024). sourceExperimental — accuracy not yet measured
Citation 10 is attached to the claim about simplified evaluations such as multiple-choice question answering. The User Experience section describes a preliminary evaluation with five researchers writing literature reviews, but our copy does not characterize literature-synthesis evaluations as typically using multiple-choice questions. Searches for that task and for evaluation benchmarks found no supporting passage; no page numbers are available in our copy. (LLM, conf=0.72)
- Partly supportedcites [9] Zheng, Y. et al. OpenResearcher: unleashing AI for accelerated scientific research. In Proc. 2024 Conf. Empirical Methods in Natural Language Processing: System Demonstrations (eds Hernández Farías, D. I., Hope, T. & Li, M.) 209–218 (Association for Computational Linguistics, 2024). sourceExperimental — accuracy not yet measured
Section 2.1 describes early scientific question-answering datasets as cloze-style tasks with predefined question formats, which supports the narrower point that some tasks were simplified. It does not identify multiple-choice question answering or say that literature-synthesis evaluations typically focus on it. Section 5.1 instead describes the paper’s own evaluation questions as complex and lengthy. (LLM, conf=0.74)
- Supportedcites [10] Skarlinski, M. D. et al. Language agents achieve superhuman synthesis of scientific knowledge. Preprint at https://arxiv.org/abs/2409.13740 (2024). source
Citation 10 is attached to the multiple-choice example, not the claim about narrow, single-discipline studies. Section 2, “Answering scientific questions” (page unavailable), describes LitQA2 as an evaluation of scientific-literature retrieval using 248 multiple-choice questions. The paper therefore supports its use as an example of that evaluation format. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Feeding a large number of documents that might include irrelevant content to LLMs can cause efficiency and performance issues, even with state-of-the-art models31,32.”
§ Synthesizing scientific literature with retrieval-augmented language models > Methods > OpenScholar
What the sources say
- Supportedcites [31] Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2024). sourceExperimental — accuracy not yet measured
Section 5 reports that adding more than 20 retrieved documents brings only marginal performance gains while significantly increasing context length, latency, and cost. Section 2.3 reports substantial performance drops when relevant information appears in the middle of a multi-document context; Appendix E notes that this effect persists for GPT-4 despite its higher overall performance. (LLM, conf=0.92)
- Not matchedcites [32] Xu, F., Shi, W. & Choi, E. RECOMP: improving retrieval-augmented LMs with compression and selective augmentation. In Proc. 12th Int. Conf. Learning Representations (Vienna, 2024; https://iclr.cc/Conferences/2024).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To overcome the difficulties of evaluating long-form, comprehensive responses13–16, ScholarQABench introduces a rigorous evaluation protocol combining automatic metrics (for example, citation accuracy) with human rubric-based assessments of coverage, coherence, writing quality and factual correctness to enable reliable assessments of the detailed long-form answers of LMs.”
§ Synthesizing scientific literature with retrieval-augmented language models > Main
What the sources say
- Supportedcites [13] Xu, F. et al. KIWI: a dataset of knowledge-intensive writing instructions for answering research questions. In Findings of the Association for Computational Linguistics: ACL 2024 (eds Ku, L.-W., Martins, A. & Srikumar, V.) 12969–12990 (ACL, 2024). source
The citation is attached to the difficulty of evaluating long-form responses, not to ScholarQABench’s own protocol. In “LLM-based Evaluation,” KIWI reports that LLMs struggle to evaluate whether responses follow precise instructions; §5 applies that evaluation task to revisions of long-form answers. The source is not being cited for ScholarQABench’s particular metrics or human rubric. (LLM, conf=0.86)
- Supportedcites [14] Xu, F., Song, Y., Iyyer, M. & Choi, E. A critical evaluation of evaluations for long-form question answering. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Rogers, A., Boyd-Graber, J. & Okazaki, N.) 3225–3245 (ACL, 2023). source
The citation is attached to the difficulties of evaluating long-form responses, not to the description of ScholarQABench’s protocol. The Abstract states that the flexibility of long-form question answering poses enormous challenges for evaluation; this supports the cited point. (LLM, conf=0.92)
- Not matchedcites [15] Kamoda, G., Asai, A., Brassard, A. & Sakaguchi, K. Quantifying the influence of evaluation aspects on long-form response assessment. In Proc. 31st Int. Conf. Computational Linguistics (eds Rambow, O. et al.) 8787–8808 (ACL, 2025).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [16] Krishna, K., Roy, A. & Iyyer, M. Hurdles to progress in long-form question answering. In Proc. 2021 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Toutanova, K. et al.) 4940–4957 (ACL, 2021). source
The citation is attached to the difficulties of evaluating long-form responses, not to ScholarQABench’s protocol. The paper’s Takeaway on automatic metrics says ROUGE-L is not meaningful for this task and human evaluation is necessary; Section 3.4 says human evaluation is challenging. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“PubMedQA41 has expert-annotated (yes/no/maybe) QA data on PubMed paper abstracts.”
§ Synthesizing scientific literature with retrieval-augmented language models > Methods > Single-paper tasks
What the sources say
- Supportedcites [41] Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. In Proc. 2019 Conf. Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP) (eds Inui, K., Jiang, J., Ng, V. & Wan, X.) 2567–2577 (ACL, 2019). source
The Abstract (p. 1) identifies PubMedQA as a dataset collected from PubMed abstracts, says its task is to answer questions with yes/no/maybe, and reports 1,000 expert-annotated QA instances. Section 3.1 (p. 3) clarifies that the expert-annotated data are one subset, alongside unlabeled and artificially generated subsets. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
56 more claims · 2 with their cited sources checked
Create a free account to see all 158 claims and to verify the 4 references still open.
- It may state two separate claims as one. 74
- Probably not a substantial claim. 31
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have shown that even state-of-the-art models hallucinate references at rates of 18-90% in long-form scientific text [8, 9]”
From Citation Hallucination Determines Success: An Empirical Comparison of Six Medical AI Research Systems (2026) · cites this paper as [9]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Synthesizing scientific literature with retrieval-augmented language models > Main
“ScholarQABench consists of 2,200 expert-written questions across several scientific disciplines”
§ Synthesizing scientific literature with retrieval-augmented language models > Abstract
“comprising 2,967 expert-written queries and 208 long-form answers”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Synthesizing scientific literature with retrieval-augmented language models > Methods > Single-paper tasks
“We collected 100 questions along with detailed answer rubrics for each question”
§ Synthesizing scientific literature with retrieval-augmented language models > Limitations > Limitations of ScholarQABench
“our human-written evaluation sets are small (for example, 110 for computer science long-form question answering; 108 expert answers)”
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“Recently, several works show that powerful general-purpose LLMs can also show strong capabilities in scientific tasks, such as medical question answering 56,67 , chemistry experimentation 68 and applied mechanics 69 .”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
Self-citationLost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 1×
“Feeding a large number of documents that might include irrelevant content to LLMs can cause efficiency and performance issues, even with state-of-the-art models 31,32 .”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · cited 1×
“Scientific LMs. Scientific LMs have spanned various domains, including biomedical 53–55 , medical 56–59 , biomedical 60–62 , geoscience 63 and astronomy 64 , with some models such as SciGLM 65 and Uni-SMART 66 that aim to cover diverse scientific domains in a single model.”
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · cited 1×
“For single-paper tasks, we directly compare the model outputs to gold reference texts, following the evaluation methodologies proposed in previous work 41–43 .”
Nogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523 · cited 2×
“To overcome these challenges, we use a cross-encoder reranker 33,34 , denoted as θ cross .”
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2023) · arXiv · doi:10.48550/arxiv.2309.07597 · Cited by 514
Research works printed but not linked: 17
- Mishra, A. et al. Fine-grained hallucination detection and editing for language models. In Proc. 1st Conf. Language Modeling (Philadelphia, 2024; https://colmweb.org).
- Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proc. 34th Int. Conf. Neural Information Processing Systems (NIPS ’20) (eds Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H.) 9459–9474 (Curran Associates, 2020).
- Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M. Retrieval augmented language model pre-training. In Proc. 37th International Conference on Machine Learning (eds Daumé, H. & Singh, A.) 3929–3938 (PMLR, 2020).
- Asai, A., Wu, Z., Wang, Y., Sil, A. & Hajishirzi, H. Self-RAG: learning to retrieve, generate, and critique through self-reflection. In Proc. 12th Int. Conf. Learning Representations (eds Kim. B et al.) (2024).
- Wang, X. et al. Scibench: evaluating college-level scientific problem-solving abilities of large language models. In Proc. 41st Int. Conf. Machine Learning (eds Salakhutdinov, R. et al.) 50622–50649 (PMLR, 2024).
- Kamoda, G., Asai, A., Brassard, A. & Sakaguchi, K. Quantifying the influence of evaluation aspects on long-form response assessment. In Proc. 31st Int. Conf. Computational Linguistics (eds Rambow, O. et al.) 8787–8808 (ACL, 2025).
- Kandpal, N., Deng, H., Roberts, A., Wallace, E. & Raffel, C. Large language models struggle to learn long-tail knowledge. In Proc. 40th International Conference on Machine Learning (eds Krause, A. et al.) 15696–15707 (PMLR, 2023).
- Dubois, Y., Galambosi, B., Liang, P. & Hashimoto, T. B. Length-controlled alpacaeval: a simple way to debias automatic evaluators. In Proc. 1st Conf. Language Modeling (Philadelphia, 2025; https://colmweb.org).
- Thakur, N., Reimers, N., Rücklé, A., Srivastava, A. & Gurevych, I. BEIR: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021) (eds Vanschoren, J. & Yeung, S.) (Curran Associates, 2021).
- Izacard, G. et al. Unsupervised dense information retrieval with contrastive learning. Trans. Machine Learning Res. https://api.semanticscholar.org/CorpusID:249097975 (2022).
- Xu, F., Shi, W. & Choi, E. RECOMP: improving retrieval-augmented LMs with compression and selective augmentation. In Proc. 12th Int. Conf. Learning Representations (Vienna, 2024; https://iclr.cc/Conferences/2024).
- Lee, Y. et al. QASA: advanced question answering on scientific articles. In Proc. 40th Int. Conf. Machine Learning (eds Krause, A. et al.) 19036–19052 (PMLR, 2023).
- Kim, S. et al. PROMETHEUS: inducing fine-grained evaluation capability in language models. In Proc. 12th International Conference on Learning Representations (eds Kim. B et al.) (2024).
- Si, C., Yang, D. & Hashimoto, T. Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers. In Proc. 13th Int. Conf. Learning Representations (Singapore, 2025; https://iclr.cc/Conferences/2025).
- Rein, D. et al. GPQA: a graduate-level Google-proof Q&A benchmark. In Proc. 1st Conf. Language Modeling (Philadelphia, 2024; https://colmweb.org).
- Chiang, W. L. et al. Chatbot arena: an open platform for evaluating llms by human preference. In Proc. 41st International Conference on Machine Learning (eds Salakhutdinov, R. et al.) 8359–8388 (PMLR, 2024).
- Huang, Q., Vora, J., Liang, P. & Leskovec, J. MLAgentBench: evaluating language agents on machine learning experimentation. In Proc. 41st Int. Conf. Machine Learning (eds Salakhutdinov, R. et al.) 20271–20309 (PMLR, 2024).
Cited by
Resnik, D. B., & Hosseini, M. (2026) · Accountability in Research · doi:10.1080/08989621.2026.2645390 · Cited by 26
“The problem of GenAI hallucinated citations is unlikely to go away completely because it is inextricably linked to how LLMs operate (Asai et al. 2026; Jones 2025; Kalai et al. 2025; Open AI 2025).”
Кузнецов, И. И. (2026) · Modelirovanie, optimizaciâ i informacionnye tehnologii. · doi:10.26102/2310-6018/2026.57.6.020 · Cited by 2
Navigating the promise and pitfalls of artificial intelligence
(2026) · Nature Microbiology · doi:10.1038/s41564-026-02300-y · Cited by 2
Open-source AI tool beats giant LLMs in literature reviews — and gets citations right
Gibney, E. (2026) · Nature · doi:10.1038/d41586-026-00347-9 · Cited by 2
LLM hallucinations in the wild: Large-scale evidence from non-existent citations
Zhao, Z., Wang, Y., Stuart, T., De Vaan, M., Ginsparg, P., & Yin, Y. (2026) · arXiv · doi:10.48550/arxiv.2605.07723 · Cited by 3
“Following existing practices in citation hallucination detection [21], we focus on one specific type of hallucinated reference: those with non-existent titles.”
Metadata
- Authors
- Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., Wadden, D., Latzke, M., Sparks, J., Hwang, J. D., Kishore, V., Tian, M., Ji, P., Liu, S., Tong, H., Wu, B., Xiong, Y., Zettlemoyer, L., Neubig, G., Weld, D. S., Downey, D., Yih, W.-T., Koh, P. W., & Hajishirzi, H.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- University of Illinois Urbana-Champaign (US)
- University of North Carolina at Chapel Hill (US)
- Stanford University (US)
- Carnegie Mellon University (US)
- Corresponding author
- Hajishirzi, Hannaneh
- ORCID
- Venue
- Nature
- Published
- 2026-02-04
- Volume
- 650
- Issue
- 8103
- Pages
- 857-863
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC12935541
- OpenAlex
- W7127589252
- Semantic Scholar
- de28e4cbee455fba136072a0f922c49698b0fb58
- Cited by
- 38
- References
- 44
- Citations by year
- 2026: 37
- FWCI
- 124.54
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion, cc-by-nc-nd)
- UNC Libraries
- Funders
- National Research Foundation
- National Research Foundation Singapore
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- retrieval-augmented generation, large language models, citation accuracy, hallucinations, GPT-4o
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Periodicals as Topic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Synthesizing scientific literature with retrieval-augmented language models. https://app.cochat.ai/papers/10.1038/s41586-025-10072-4
- The paper
- Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., Wadden, D., Latzke, M., Sparks, J., Hwang, J. D., Kishore, V., Tian, M., Ji, P., Liu, S., Tong, H., . . . Hajishirzi, H. (2026). Synthesizing scientific literature with retrieval-augmented language models. Nature. https://doi.org/10.1038/s41586-025-10072-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.