ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition
Open accessAbstract
Large language models (LLMs) have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. To address this gap, we introduce the first large-scale benchmark for evaluating LLMs on a sufficient set of scientific discovery sub-tasks-inspiration retrieval, hypothesis composition, and hypothesis ranking-where sufficient means that perfectly solving these sub-tasks perfectly solves the overall discovery task. We develop an automated LLM-based framework that extracts critical components-research questions, background surveys, inspirations, and hypotheses-from papers across 12 disciplines, with expert validation confirming its accuracy. To prevent data contamination, we focus exclusively on publications from 2024 onward, ensuring minimal overlap with LLM pretraining data; our automated framework further enables automatic extraction of even more recent papers as LLM pretraining cutoffs advance, supporting scalable and contamination-free automatic renewal of this discovery benchmark. Our evaluation shows that, across disciplines, LLMs excel at inspiration retrieval-an out-of-distribution task-suggesting their ability to surface novel knowledge associations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing benchmarks face two primary limitations: reliance on static datasets or answer keys, and use of non-expert or automated evaluation including coarse metrics or auto-generated rubrics.
“To summarize, existing benchmarks thus face two primary limitations: a reliance on static datasets or answer keys [15, 16], and the use of non-expert or automated evaluation, including coarse metrics [20] or auto-generated rubrics [8, 11, 24].”
From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [16]
Section 4.2 describes scoring generated hypotheses against ground-truth hypotheses using a six-point scale, supporting the characterization of automated evaluation. Appendix A.4 provides fixed scoring criteria; our copy does not say the rubrics are auto-generated. Section 3.4 also reports expert evaluation of the benchmark’s extracted components, so the source should not be characterized as relying solely on non-expert evaluation. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 21
- Mathias Benedek, Tanja Könen, and Aljoscha C Neubauer. 2012. Associative abilities underlying creativity. Psychology of Aesthetics, Creativity, and the Arts, 6(3):273.
- Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anas- tasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chat- bot arena: An open platform for evaluating llms by human preference. In Forty-first International Con- ference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
- Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. · arXiv
- Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407.
- Aniketh Garikaparthi, Manasi Patwardhan, Aditya San- jiv Kanade, Aman Hassan, Lovekesh Vig, and Ar- man Cohan. 2025. Mir: Methodology inspiration retrieval for scientific research problems. arXiv preprint arXiv:2506.00249. · arXiv
- Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, and Aidong Zhang. 2024. Ideabench: Benchmarking large language models for research idea generation. CoRR, abs/2411.02429. · DOI
- Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. · arXiv
- Byung Cheol Lee and Jaeyeon Chung. 2024. An empiri- cal investigation of the impact of chatgpt on creativity. Nature Human Behaviour, pages 1-9.
- Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. · arXiv
- Wanhao Liu, Zonglin Yang, Jue Wang, Lidong Bing, Di Zhang, Dongzhan Zhou, Yuqiang Li, Houqiang Li, Erik Cambria, and Wanli Ouyang. 2025. Moose- chem3: Toward experiment-guided hypothesis rank- ing via simulated experimental feedback. arXiv preprint arXiv:2505.17873. · arXiv
- Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. 2025. Llm4sr: A survey on large lan- guage models for scientific research. arXiv preprint arXiv:2501.04306. · arXiv
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Itera- tive refinement with self-feedback. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565. · arXiv
- Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Fran- cisco JR Ruiz, Abbas Mehrabian, et al. 2025. Alphae- volve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. · arXiv
- Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468-475.
- Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. 2025a. Llm-sr: Scientific equation discovery via program- ming with large language models. In The Thirteenth International Conference on Learning Representa- tions.
- Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. 2025b. Llm-srbench: A new benchmark for scientific equation discovery with large language models. arXiv preprint arXiv:2504.10415. · arXiv
- Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large- scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109. · arXiv
- Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thomp- son, et al. 2025. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01848. · arXiv
- Qwen Team et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2:3. · arXiv
- Yixuan Weng, Minjun Zhu, Qiujie Xie, Qiyao Sun, Zhen Lin, Sifan Liu, and Yue Zhang. 2025. Deepscientist: Advancing frontier-pushing scientific findings pro- gressively. arXiv preprint arXiv:2509.26603. · arXiv
Cited by
Sorted by citations
CellVoyager: AI CompBio agent generates new insights by autonomously analyzing biological data
Alber, S., Chen, B., Sun, E., Isakova, A., Wilk, A. J., & Zou, J. (2026) · Nature Methods · doi:10.1038/s41592-026-03029-6 · Cited by 40 · see citing works
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
“Notably, recent evaluations by Gu et al. (2024); Kumar et al. (2024); Liu et al. (2025); Si et al. (2024) suggest that AI-generated research concepts may occasionally exceed human-generated ideas in terms of novelty and originality.”
ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Sharma, M., Zhang, C. B. C., Bandi, C., Wang, C., Aich, A., Nghiem, H., Rabbani, T., Htet, Y., Jang, B., Basu, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.07685 · Cited by 1 · see citing works
“Derived Benchmarks AcademicBrowse [30] and BrowseComp [26] assess retrieval from academic papers or the web, while ResearchBench [16] builds complex queries from static data.”
Harnessing Large Language Models for Scientific Novelty Detection
Liu, Y., Yang, Z., Poria, S., Nguyen, T.-S., & Cambria, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_39
Self-citation
- Authors
- Liu, Y., Yang, Z., Xie, T., Ni, J., Gao, B., Li, Y., Tang, S., Ouyang, W., Cambria, E., & Zhou, D.
- Affiliations
- Fudan University (CN)
- Nanyang Technological University (SG)
- National University of Singapore (SG)
- Wuhan University (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Pages
- 13187-13207
- arXiv
- 2503.21248
- OpenAlex
- W4415062263
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 2
- FWCI
- 17.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Institute for Catastrophic Loss Reduction
- National Science and Technology Major Project (2025ZD0121802)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, scientific discovery, research hypothesis generation, LLM benchmark, hypothesis ranking, task decomposition, contaminated data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
