Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; 27 of the 43 sources its claims cite were checked for support.

Synthesizing scientific literature with retrieval-augmented language models

Open access

Abstract

Scientific progress depends on the ability of researchers to synthesize the growing body of literature. Can large language models (LLMs) assist scientists in this task? Here we introduce OpenScholar, a specialized retrieval-augmented language model (LM)1 that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience and biomedicine. Despite being a smaller open model, OpenScholar-8B outperforms GPT-4o by 6.1% and PaperQA2 by 5.5% in correctness on a challenging multi-paper synthesis task from the new ScholarQABench. Although GPT-4o hallucinates citations 78–90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar’s data store, retriever and self-feedback inference loop improve off-the-shelf LMs: for instance, OpenScholar-GPT-4o improves the correctness of GPT-4o by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT-4o responses over expert-written ones 51% and 70% of the time, respectively, compared with 32% for GPT-4o. We open-source all artefacts, including our code, models, data store, datasets and a public demo. A specialized, open-source, retrieval-augmented language model is introduced for answering scientific queries and synthesizing literature, the responses of which are shown to be preferred by human evaluations over expert-written answers.

Overview

CoChat Summary

TL;DR

OpenScholar is a fully open retrieval-augmented language model that answers scientific queries by retrieving passages from 45 million open-access papers and synthesizing citation-backed responses. On the new ScholarQABench, OpenScholar-8B outperforms GPT-4o and PaperQA2 on multi-paper synthesis correctness, and OpenScholar variants are often preferred over expert-written answers in human evaluations.

Key findings

  • OpenScholar-8B outperforms GPT-4o by 6.1% and PaperQA2 by 5.5% in correctness on a challenging multi-paper synthesis task from ScholarQABench; OpenScholar-GPT-4o improves GPT-4o correctness by 12%.
  • GPT-4o hallucinates citations 78–90% of the time, whereas OpenScholar achieves citation accuracy on par with human experts; in Table 2, OpenScholar-8B had 0 hallucinated papers in computer science and biomedicine, versus 78.1–97.6% hallucination ratios for Llama 3.1 and GPT-4o without retrieval.
  • In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT-4o responses over expert-written ones 51% and 70% of the time, respectively, compared with 32% for GPT-4o.
  • On Scholar-CS rubric scores, OpenScholar-GPT-4o scored 57.7, OpenScholar-8B 51.1, human experts 54.0, PaperQA2 45.6 and GPT-4o 45.0; OpenScholar-8B also outperformed PaperQA2 and Perplexity Pro by 6% and 10%, respectively.

Methods

  • OpenScholar combines a domain-specialized data store (OSDS) of 45 million scientific papers and 236 million passage embeddings, trained retrievers/rerankers, citation-aware generation, and a self-feedback iterative RAG loop with citation verification.
  • ScholarQABench was built with 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience and biomedicine; it includes single-paper tasks (PubMedQA, SciFact, QASA) and multi-paper long-form tasks (Scholar-CS, Scholar-Multi, Scholar-Bio, Scholar-Neuro).
  • Evaluation used automatic metrics (correctness, citation F1, LLM-judge scores) plus human rubric-based assessments; expert evaluations involved 16 PhD-level raters and 108 questions from Scholar-Multi.

Limitations

  • The paper states that expert annotation is costly, so human-written evaluation sets are small (for example, 110 for computer science long-form QA; 108 expert answers), which may introduce variance and annotator-expertise bias.
  • The paper states that automatic evaluation may not perfectly capture quality: Scholar-CS combines length, excerpts and rubric items with heuristic weights; citation precision/recall is sentence-level and can be overly strict; annotations reflect specific time points (July 2024 for Scholar-CS, September 2024 for Scholar-Multi); and the static public benchmark raises future contamination risks.
  • The paper states that ScholarQABench primarily covers computer science, biomedicine and physics, with no social sciences or other engineering/scientific disciplines, so findings may not fully generalize; OpenScholar also does not consistently retrieve the most representative papers, may contain factual inaccuracies or unsupported information (especially the 8B model), and OpenScholar-GPT-4o relies on the proprietary GPT-4o API, making exact replication challenging.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-025-10072-4

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Overview of OpenScholar, ScholarQABench and evaluation results. Top, overview of OpenScholar. OpenScholar consists of a specialized data store (OSDS), retrievers and LMs and iteratively improves responses using self-feedback inference with retrieval. Middle, overview of ScholarQABench. ScholarQABench consists of 2,200 expert-written questions across several scientific disciplines and we introduce automatic and human evaluation protocols for ScholarQABench. Bottom, automatic and human evaluation results: experimental results from the ScholarQABench computer science subset (Scholar-CS, 100 questions) show that OpenScholar with our trained 8B or GPT-4o substantially outperforms other systems and is preferred over experts more than 50% of the time in human evaluations. Our human evaluations were conducted by 16 experts with PhDs across 108 questions from Scholar-Multi.
Fig. 1 Overview of OpenScholar, ScholarQABench and evaluation results. Top, overview of OpenScholar. OpenScholar consists of a specialized data store (OSDS), retrievers and LMs and iteratively improves responses using self-feedback inference with retrieval. Middle, overview of ScholarQABench. ScholarQABench consists of 2,200 expert-written questions across several scientific disciplines and we introduce automatic and human evaluation protocols for ScholarQABench. Bottom, automatic and human evaluation results: experimental results from the ScholarQABench computer science subset (Scholar-CS, 100 questions) show that OpenScholar with our trained 8B or GPT-4o substantially outperforms other systems and is preferred over experts more than 50% of the time in human evaluations. Our human evaluations were conducted by 16 experts with PhDs across 108 questions from Scholar-Multi.CC BY-NC-ND · PubMed Central
Extended Data Fig. 1 Detailed overview of the OpenScholar inference (top) and training pipeline (bottom). At inference time, given an input x, OpenScholar first uses a retriever to identify relevant papers from a specialized data store (OSDS) and then uses a reranker to refine and identify the top N retrieved documents. The retrieved output is then passed to the LM, which generates both (1) an initial response y0 and (2) self-feedback f1. By incorporating its feedback, the LM iteratively refines its output a pre-defined number of times. Subsequently, a LM (1) generates an initial response y0, (2) generates self-feedback on the initial output and (3) incorporates feedback (fi) to generate a revised response y1. The LM repeats the process until all feedback is incorporated. To train a smaller yet competitive 8B LM, we generate high-quality training data using this inference-time pipeline, followed by data filtering and mixing. We use our new OpenScholar retriever, continue pre-training on the OSDS for the retriever, followed by OpenScholar reranker, fine-tuned on synthetically generated data initialized from the BGE reranker, for the reranker component. For OpenScholar-8B, we use Llama 3.1 8B as the generator and for OpenScholar-GPT-4o, we use GPT-4o as the LM.
Extended Data Fig. 1 Detailed overview of the OpenScholar inference (top) and training pipeline (bottom). At inference time, given an input x, OpenScholar first uses a retriever to identify relevant papers from a specialized data store (OSDS) and then uses a reranker to refine and identify the top N retrieved documents. The retrieved output is then passed to the LM, which generates both (1) an initial response y0 and (2) self-feedback f1. By incorporating its feedback, the LM iteratively refines its output a pre-defined number of times. Subsequently, a LM (1) generates an initial response y0, (2) generates self-feedback on the initial output and (3) incorporates feedback (fi) to generate a revised response y1. The LM repeats the process until all feedback is incorporated. To train a smaller yet competitive 8B LM, we generate high-quality training data using this inference-time pipeline, followed by data filtering and mixing. We use our new OpenScholar retriever, continue pre-training on the OSDS for the retriever, followed by OpenScholar reranker, fine-tuned on synthetically generated data initialized from the BGE reranker, for the reranker component. For OpenScholar-8B, we use Llama 3.1 8B as the generator and for OpenScholar-GPT-4o, we use GPT-4o as the LM.CC BY-NC-ND · PubMed Central
Extended Data Fig. 2 Example Scholar-CS question, rubric and evaluation pipeline. Scholar-CS consists of 100 questions and an average of 4.4 expert-written rubrics to be satisfied. Our ScholarQABench evaluation pipeline evaluates aspects such as correctness and citation accuracy.
Extended Data Fig. 2 Example Scholar-CS question, rubric and evaluation pipeline. Scholar-CS consists of 100 questions and an average of 4.4 expert-written rubrics to be satisfied. Our ScholarQABench evaluation pipeline evaluates aspects such as correctness and citation accuracy.CC BY-NC-ND · PubMed Central
Extended Data Fig. 3 Effect of context length on Scholar-CS correctness. Analysis of how varying the number of retrieved passages (top N) affects rubric scores (correctness) for standard RAG and OpenScholar using Llama 3.1 8B and our trained 8B model.
Extended Data Fig. 3 Effect of context length on Scholar-CS correctness. Analysis of how varying the number of retrieved passages (top N) affects rubric scores (correctness) for standard RAG and OpenScholar using Llama 3.1 8B and our trained 8B model.CC BY-NC-ND · PubMed Central
Extended Data Fig. 4 Effect of context length. Analysis of how varying the number of retrieved passages (top N) affects citation F1 for standard RAG and OpenScholar using Llama 3.1 8B and our trained 8B model.
Extended Data Fig. 4 Effect of context length. Analysis of how varying the number of retrieved passages (top N) affects citation F1 for standard RAG and OpenScholar using Llama 3.1 8B and our trained 8B model.CC BY-NC-ND · PubMed Central
Extended Data Fig. 5 Fine-grained expert evaluations of human and model-written answers. Score distributions from expert raters comparing expert-written answers with GPT-4o, OpenScholar-8B and OpenScholar-GPT-4o on Scholar-Multi. The panels show histograms for organization, coverage and relevance on a five-point scale, as well as relative win/tie/lose rates against human answers. OpenScholar-GPT-4o and OpenScholar-8B are preferred over expert answers in most cases, largely because of higher coverage and depth, whereas GPT-4o without retrieval exhibits limited coverage and lower overall usefulness despite strong fluency.
Extended Data Fig. 5 Fine-grained expert evaluations of human and model-written answers. Score distributions from expert raters comparing expert-written answers with GPT-4o, OpenScholar-8B and OpenScholar-GPT-4o on Scholar-Multi. The panels show histograms for organization, coverage and relevance on a five-point scale, as well as relative win/tie/lose rates against human answers. OpenScholar-GPT-4o and OpenScholar-8B are preferred over expert answers in most cases, largely because of higher coverage and depth, whereas GPT-4o without retrieval exhibits limited coverage and lower overall usefulness despite strong fluency.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

15 Valid · 11 Not valid · 8 Not checked · 124 Own results and interpretations

56 more claims · 2 with their cited sources checked

Create a free account to see all 158 claims and to verify the 4 references still open.

See all claims — free account
  • It may state two separate claims as one. 74
  • Probably not a substantial claim. 31

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 44 references

Research works printed but not linked: 17

  1. Mishra, A. et al. Fine-grained hallucination detection and editing for language models. In Proc. 1st Conf. Language Modeling (Philadelphia, 2024; https://colmweb.org).
  2. Lewis, P. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proc. 34th Int. Conf. Neural Information Processing Systems (NIPS ’20) (eds Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H.) 9459–9474 (Curran Associates, 2020).
  3. Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M. Retrieval augmented language model pre-training. In Proc. 37th International Conference on Machine Learning (eds Daumé, H. & Singh, A.) 3929–3938 (PMLR, 2020).
  4. Asai, A., Wu, Z., Wang, Y., Sil, A. & Hajishirzi, H. Self-RAG: learning to retrieve, generate, and critique through self-reflection. In Proc. 12th Int. Conf. Learning Representations (eds Kim. B et al.) (2024).
  5. Wang, X. et al. Scibench: evaluating college-level scientific problem-solving abilities of large language models. In Proc. 41st Int. Conf. Machine Learning (eds Salakhutdinov, R. et al.) 50622–50649 (PMLR, 2024).
  6. Kamoda, G., Asai, A., Brassard, A. & Sakaguchi, K. Quantifying the influence of evaluation aspects on long-form response assessment. In Proc. 31st Int. Conf. Computational Linguistics (eds Rambow, O. et al.) 8787–8808 (ACL, 2025).
  7. Kandpal, N., Deng, H., Roberts, A., Wallace, E. & Raffel, C. Large language models struggle to learn long-tail knowledge. In Proc. 40th International Conference on Machine Learning (eds Krause, A. et al.) 15696–15707 (PMLR, 2023).
  8. Dubois, Y., Galambosi, B., Liang, P. & Hashimoto, T. B. Length-controlled alpacaeval: a simple way to debias automatic evaluators. In Proc. 1st Conf. Language Modeling (Philadelphia, 2025; https://colmweb.org).
  9. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A. & Gurevych, I. BEIR: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021) (eds Vanschoren, J. & Yeung, S.) (Curran Associates, 2021).
  10. Izacard, G. et al. Unsupervised dense information retrieval with contrastive learning. Trans. Machine Learning Res. https://api.semanticscholar.org/CorpusID:249097975 (2022).
  11. Xu, F., Shi, W. & Choi, E. RECOMP: improving retrieval-augmented LMs with compression and selective augmentation. In Proc. 12th Int. Conf. Learning Representations (Vienna, 2024; https://iclr.cc/Conferences/2024).
  12. Lee, Y. et al. QASA: advanced question answering on scientific articles. In Proc. 40th Int. Conf. Machine Learning (eds Krause, A. et al.) 19036–19052 (PMLR, 2023).
  13. Kim, S. et al. PROMETHEUS: inducing fine-grained evaluation capability in language models. In Proc. 12th International Conference on Learning Representations (eds Kim. B et al.) (2024).
  14. Si, C., Yang, D. & Hashimoto, T. Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers. In Proc. 13th Int. Conf. Learning Representations (Singapore, 2025; https://iclr.cc/Conferences/2025).
  15. Rein, D. et al. GPQA: a graduate-level Google-proof Q&A benchmark. In Proc. 1st Conf. Language Modeling (Philadelphia, 2024; https://colmweb.org).
  16. Chiang, W. L. et al. Chatbot arena: an open platform for evaluating llms by human preference. In Proc. 41st International Conference on Machine Learning (eds Salakhutdinov, R. et al.) 8359–8388 (PMLR, 2024).
  17. Huang, Q., Vora, J., Liang, P. & Leskovec, J. MLAgentBench: evaluating language agents on machine learning experimentation. In Proc. 41st Int. Conf. Machine Learning (eds Salakhutdinov, R. et al.) 20271–20309 (PMLR, 2024).

Cited by

See all 38

Metadata

Authors
Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., Wadden, D., Latzke, M., Sparks, J., Hwang, J. D., Kishore, V., Tian, M., Ji, P., Liu, S., Tong, H., Wu, B., Xiong, Y., Zettlemoyer, L., Neubig, G., Weld, D. S., Downey, D., Yih, W.-T., Koh, P. W., & Hajishirzi, H.
Affiliations
  • University of Washington (US)
  • Allen Institute for Artificial Intelligence (US)
  • University of Illinois Urbana-Champaign (US)
  • University of North Carolina at Chapel Hill (US)
  • Stanford University (US)
  • Carnegie Mellon University (US)
Corresponding author
Hajishirzi, Hannaneh
Venue
Nature
Published
2026-02-04
Volume
650
Issue
8103
Pages
857-863
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W7127589252
Cited by
38
References
44
Citations by year
2026: 37
FWCI
124.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion, cc-by-nc-nd)
  • UNC Libraries
Funders
  • National Research Foundation
  • National Research Foundation Singapore
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
retrieval-augmented generation, large language models, citation accuracy, hallucinations, GPT-4o
MeSH terms
Large Language Models, Humans, Natural Language Processing, Periodicals as Topic, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Synthesizing scientific literature with retrieval-augmented language models. https://app.cochat.ai/papers/10.1038/s41586-025-10072-4
The paper
Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., Wadden, D., Latzke, M., Sparks, J., Hwang, J. D., Kishore, V., Tian, M., Ji, P., Liu, S., Tong, H., . . . Hajishirzi, H. (2026). Synthesizing scientific literature with retrieval-augmented language models. Nature. https://doi.org/10.1038/s41586-025-10072-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.