CoChat
Sign Up
Cited by 9Topic ModelingArtificial Intelligence ApplicationsMultimodal Machine Learning Applications

IdeaBench: Benchmarking Large Language Models for Research Idea Generation

Open access
Guo, Sikun, Shariatmadari, Amir Hassan, Xiong, Guangzhi, Huang, Albert, Kim, Myles, Williams, Corey M., Bekiranov, Stefan, Zhang, Aidong·2025·Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
DOI 10.1145/3711896.3737419
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) have revolutionized interactions between human and artificial intelligence (AI) systems, demonstrating state-of-the-art performance across various domains, including scientific discovery and hypothesis generation. However, the absence of a comprehensive and systematic evaluation framework for LLM-driven research idea generation hinders a rigorous understanding of their strengths and limitations. To address this gap, we propose IdeaBench, a benchmark system that provides a structured dataset and evaluation framework for standardizing the assessment of research idea generation by LLMs. Our dataset comprises titles and abstracts from 2,374 influential papers across eight research domains, along with their 29,408 referenced works, creating a context-rich environment that mirrors human researchers' ideation processes. By profiling LLMs as domain-specific researchers and grounding them in similar contextual constraints, we directly leverage the models' knowledge learned from the pre-training stage to generate new research ideas. To systematically evaluate LLMs' research ideation capability and approximate human assessment, we propose a reference-based metric that aligns with human judgment to quantify idea quality with the assistance of LLMs. Through this evaluation, we find that while LLMs excel at generating novel ideas, they may struggle with generating feasible ideas. IdeaBench serves as a critical resource for benchmarking and comparing LLMs, ultimately advancing research on AI's role in automating scientific discovery.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3711896.3737419

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLM systems perform as well as or even better than humans on commonly accepted creative ideation benchmarks and on benchmarks for ideation in specific contexts.

    “The first is that LLM systems perform as well as or even better than humans on commonly accepted creative ideation benchmarks [8] and benchmarks for ideation in specific contexts [7].”

    From The Role of Human Agency in Human-AI Co-Creativity (2026) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. AI4Science, M. R.; and Quantum, M. A. 2023. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361 . · arXiv
  2. Baek, J.; Jauhar, S. K.; Cucerzan, S.; and Hwang, S. J. 2024. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738 . · arXiv
  3. Bai, Y.; Ying, J.; Cao, Y.; Lv, X.; He, Y.; Wang, X.; Yu, J.; Zeng, K.; Xiao, Y.; Lyu, H.; et al. 2024. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems , 36.
  4. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems , 33: 18771901.
  5. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y .; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 . · arXiv
  6. Fu, J.; Ng, S.-K.; Jiang, Z.; and Liu, P. 2024. GPTScore: Evaluate as You Desire. In Duh, K.; Gomez, H.; and Bethard, S., eds., Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , 6556-6576. Mexico City, Mexico: Association for Computational Linguistics.
  7. Google Scholar. 2024. Google Scholar Top Publications. https://scholar.google.com/citations?view op=top venues.
  8. Kinney, R.; Anastasiades, C.; Authur, R.; Beltagy, I.; Bragg, J.; Buraczynski, A.; Cachola, I.; Candra, S.; Chandrasekhar, Y.; Cohan, A.; et al. 2023. The semantic scholar open data platform. arXiv preprint arXiv:2301.10140 . · arXiv
  9. Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35: 22199-22213.
  10. Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Annual Meeting of the Association for Computational Linguistics .
  11. Moore, S.; Tong, R.; Singh, A.; Liu, Z.; Hu, X.; Lu, Y.; Liang, J.; Cao, C.; Khosravi, H.; Denny, P.; et al. 2023. Empowering education with llms-the next-gen interface and content generation. In International Conference on Artificial Intelligence in Education , 32-37. Springer.
  12. OpenAI. 2023. GPT-4 Technical Report. ArXiv , abs/2303.08774. · arXiv
  13. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 311-318.
  14. Qiu, L.; Jiang, L.; Lu, X.; Sclar, M.; Pyatkin, V.; Bhagavatula, C.; Wang, B.; Kim, Y.; Choi, Y.; Dziri, N.; et al. 2023. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08559 . · arXiv
  15. Reid, M.; Savinov, N.; Teplyashin, D.; Lepikhin, D.; Lillicrap, T.; Alayrac, J.-b.; Soricut, R.; Lazaridou, A.; Firat, O.; Schrittwieser, J.; et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 . · arXiv
  16. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 . · arXiv
  17. Vu, T.; Krishna, K.; Alzubi, S.; Tar, C.; Faruqui, M.; and Sung, Y.-H. 2024. Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation. arXiv preprint arXiv:2407.10817 . · arXiv
  18. Wang, Q.; Downey, D.; Ji, H.; and Hope, T. 2023a. Learning to generate novel scientific directions with contextualized literature-based discovery. arXiv preprint arXiv:2305.14259 . · arXiv
  19. Wang, Q.; Downey, D.; Ji, H.; and Hope, T. 2023b. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259 . · arXiv
  20. Wysocki, O.; Wysocka, M.; Carvalho, D.; Bogatu, A. T.; Gusicuma, D. M.; Delmas, M.; Unsworth, H.; and Freitas, A. 2024. An LLM-based Knowledge Synthesis and Scientific Reasoning Framework for Biomedical Discovery. arXiv preprint arXiv:2406.18626 . · arXiv
  21. Yang, R.; Tan, T. F.; Lu, W.; Thirunavukarasu, A. J.; Ting, D. S. W.; and Liu, N. 2023a. Large language models in health care: Development, applications, and challenges. Health Care Science , 2(4): 255-263.
  22. Yang, Z.; Du, X.; Li, J.; Zheng, J.; Poria, S.; and Cambria, E. 2023b. Large language models for automated open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.02726 . · arXiv
  23. Yuan, W.; Neubig, G.; and Liu, P. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems , 34: 27263-27277.
  24. Zhao, W.; Peyrard, M.; Liu, F.; Gao, Y.; Meyer, C. M.; and Eger, S. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622 . · arXiv
  25. Zhou, Y.; Liu, H.; Srivastava, T.; Mei, H.; and Tan, C. 2024. Hypothesis Generation with Large Language Models. arXiv preprint arXiv:2404.04326 . · arXiv

Cited by

Sorted by citations

Authors
Guo, S., Shariatmadari, A. H., Xiong, G., Huang, A., Kim, M., Williams, C. M., Bekiranov, S., & Zhang, A.
Affiliations
University of Virginia (US)
ORCID
Venue
Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Publisher
ACM
Published
2025-08-03
Pages
5888-5899
DOI
10.1145/3711896.3737419
OpenAlex
W4412875478
Cited by
9
References
4
Citations by year
2026: 8 · 2024: 1
FWCI
7.59
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • NSF (National Science Foundation) (217071, 2213700, 2106913, 2008208)
  • NIH (National Institutes of Health) (1R01LM014012)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, research idea generation, benchmarking, scientific hypothesis generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review