The DOI matches its DataCite record.
Large Language Models for Automated Open-domain Scientific Hypotheses Discovery
Open accessAbstract
Hypothetical induction is recognized as the main reasoning type when scientists make observations about the world and try to propose hypotheses to explain those observations. Past research on hypothetical induction is under a constrained setting: (1) the observation annotations in the dataset are carefully manually handpicked sentences (resulting in a close-domain setting); and (2) the ground truth hypotheses are mostly commonsense knowledge, making the task less challenging. In this work, we tackle these problems by proposing the first dataset for social science academic hypotheses discovery, with the final goal to create systems that automatically generate valid, novel, and helpful scientific hypotheses, given only a pile of raw web corpus. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi-module framework is developed for the task, including three different feedback mechanisms to boost performance, which exhibits superior performance in terms of both GPT-4 based and expert-based evaluation. To the best of our knowledge, this is the first work showing that LLMs are able to generate novel (''not existing in literature'') and valid (''reflecting reality'') scientific hypotheses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
All prior automatic hypothesis generation approaches perform idea generation in a localized manner and are designed to identify potential relationships between two variables or generate sentence-level connections, which may be sub-optimal to capture the complexity and multifaceted nature of real-world problems.
“Yet, all these approaches perform idea generation in a localized manner and are designed to identify potential relationships between two variables or generate sentence-level connections, which may be sub-optimal to capture the complexity and multifaceted nature of real-world problems (e.g., urban planning involves numerous interacting variables).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Yang et al., 2023
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredResearchAgent generates superior research hypotheses compared to prior hypothesis generation methods (SciMON and Hypothesis Proposer).
“we observe that ResearchAgent is capable of generating superior research hypotheses”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Yang et al., 2023
Section 5.3 reports that MOOSE-base outperformed the paper’s baseline on novelty and helpfulness, but it does not evaluate ResearchAgent against SciMON or Hypothesis Proposer. Searches for ResearchAgent, SciMON, and their comparison found no passage addressing the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
- Daniil A. Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent autonomous scientific research capabilities of large language models. CoRR, abs/2304.05332. · DOI · Paper page
- Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics. · DOI · Paper page
- Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376. · arXiv
- Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya God- bole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case- based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 9594-9611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics. · DOI
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303.17651. · DOI · Paper page
- John D Norton. 2003. A little survey of induction. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774. · DOI · Paper page
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurIPS.
- Feng Pan, Rutu Mulkar-Mehta, and Jerry R. Hobbs. 2011. Annotating and learning event durations in text. Comput. Linguistics, 37(4):727-752. · DOI
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
- Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. CoRR, abs/2302.12813. · DOI · Paper page
- Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350. · DOI · Paper page
- Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Si- hang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis pro- posers. CoRR, abs/2311.05965. · DOI
- Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends® in Information Re- trieval, 3(4):333-389.
- Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic mem- ory and self-reflection. CoRR, abs/2303.11366. · DOI · Paper page
- Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023a. Scientific discovery in the age of artifi- cial intelligence. Nature, 620(7972):47-60.
- Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023b. Learning to generate novel scientific direc- tions with contextualized literature-based discovery. CoRR, abs/2305.14259. · DOI
- Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 4393-4479. Association for Computational Linguistics. · DOI
- Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024. Language models as inductive reasoners. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 209-225, St. Julian's, Malta. Association for Computational Linguistics.
- Zonglin Yang, Xinya Du, Erik Cambria, and Claire Cardie. 2023a. End-to-end case-based reasoning for commonsense knowledge base completion. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 3509-3522, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
- Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023b. Logical reasoning over natural lan- guage as knowledge representation: A survey. CoRR, abs/2303.12023. · DOI
- Zonglin Yang, Xinya Du, Alexander Rush, and Claire Cardie. 2020. Improving event duration prediction via time-aware pre-training. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 3370-3378, Online. Association for Computa- tional Linguistics. · DOI
- Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven dis- covery of distributional differences via language de- scriptions. CoRR, abs/2302.14233. · DOI · Paper page
Cited by
Sorted by citations
Baek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.342 · Cited by 46
Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions
Gridach, M., Nanavati, J., Abidine, K. Z. E., Mendes, L., & Mack, C. (2025) · arXiv · doi:10.48550/arxiv.2503.08979 · Cited by 18
AlphaEvolve: A coding agent for scientific and algorithmic discovery
Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., Shirobokov, S., Kozlovskii, B., Ruiz, F. J. R., Mehrabian, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.13131 · Cited by 15
“Many of these methods use LLMs to automate several distinct stages of the scientific discovery process [37, 59, 106, 109, 112], e.g., for generating and ranking hypotheses and ideas [38, 90].”
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5
Metadata
- Authors
- Yang, Z., Du, X., Li, J., Zheng, J., Poria, S., & Cambria, E.
- Affiliations
- Nanyang Technological University (SG)
- The University of Texas at Dallas (US)
- Huazhong University of Science and Technology (CN)
- Singapore University of Technology and Design (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 13545-13565
- arXiv
- 2309.02726
- OpenAlex
- W4402683817
- Cited by
- 34
- References
- 0
- Citations by year
- 2026: 16 · 2025: 14 · 2024: 4
- FWCI
- 10.05
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministry of Education - Singapore (STEM RIE2025, T2EP20123-0005, Tier 2)
- Ministry of Education, India
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- scientific hypothesis generation, large language models, dataset construction, feedback mechanisms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
