CoChat
Sign Up
PreprintCited by 7Artificial Intelligence ApplicationsScientific Computing and Data ManagementTopic Modeling

Automated Hypothesis Validation with Agentic Sequential Falsifications

Open access
Huang, Kexin, Jin, Ying, Li, Ryan, Li, Michael Y., Candès, Emmanuel, Leskovec, Jure·2025·arXiv
DOI 10.48550/arxiv.2502.09858arXiv 2502.09858
Open this paper in CoChat — free accountOpen PDF

Abstract

Hypotheses are central to information acquisition, decision-making, and discovery. However, many real-world hypotheses are abstract, high-level statements that are difficult to validate directly. This challenge is further intensified by the rise of hypothesis generation from Large Language Models (LLMs), which are prone to hallucination and produce hypotheses in volumes that make manual validation impractical. Here we propose Popper, an agentic framework for rigorous automated validation of free-form hypotheses. Guided by Karl Popper's principle of falsification, Popper validates a hypothesis using LLM agents that design and execute falsification experiments targeting its measurable implications. A novel sequential testing framework ensures strict Type-I error control while actively gathering evidence from diverse observations, whether drawn from existing data or newly conducted procedures. We demonstrate Popper on six domains including biology, economics, and sociology. Popper delivers robust error control, high power, and scalability. Furthermore, compared to human scientists, Popper achieved comparable performance in validating complex biological hypotheses while reducing time by 10 folds, providing a scalable, rigorous solution for hypothesis validation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.09858

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Rule-based verification methods do not generalize to complex, realistic settings where handcrafted rules are insufficient.

    “While effective for narrowly scoped tasks, such methods do not generalize to complex, realistic settings (e.g., scientific hypothesis verification or essay-level fact checking), where handcrafted rules are insufficient (Huang et al., 2025).”

    From AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation (2026) · cites this paper as Huang et al., 2025

    The Introduction says that real-world scientific hypotheses are often abstract statements that are difficult to evaluate directly and must be translated into measurable implications. That supports the example of scientific hypothesis verification as a complex task, but our copy does not establish that rule-based methods fail to generalize, that handcrafted rules are insufficient, or that the same applies to essay-level fact checking. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 7

Research works printed but not linked: 62

  1. Agassi, J. Popper and his popular critics: Thomas kuhn, paul feyerabend and imre lakatos. In SpringerBriefs in Philos- ophy. Springer, 2014. doi: 10.1007/978-3-319-06587-8. · DOI
  2. Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., and Gao, T. Litsearch: A retrieval benchmark for scientific literature search, 2024. URL https://arxiv.org/ abs/2407.18940. · arXiv
  3. Alet, F., Lopez-Contreras, J., Koppel, J., Nye, M., Solar- Lezama, A., Lozano-Perez, T., Kaelbling, L., and Tenen- baum, J. A large-scale benchmark for few-shot program induction and synthesis. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Ma- chine Learning Research, pp. 175-186. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/ v139/alet21a.html.
  4. Baek, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea generation over scientific literature with large language models, 2024. URL https://arxiv.org/abs/2404.07738. · arXiv
  5. Benjamini, Y. and Hochberg, Y. Controlling the false dis- covery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
  6. Brown, M. B. 400: A method for combining non- independent, one-sided tests of significance. Biometrics, pp. 987-992, 1975.
  7. Consortium, G. The gtex consortium atlas of genetic regu- latory effects across human tissues. Science, 369(6509): 1318-1330, 2020.
  8. D'Arcy, M., Hope, T., Birnbaum, L., and Downey, D. Marg: Multi-agent review generation for scientific papers, 2024. URL https://arxiv.org/abs/2401.04259. · arXiv
  9. Fisher, R. A. Design of experiments. British Medical Journal, 1(3923):554, 1936.
  10. Fisher, R. A. Statistical methods for research workers. In Breakthroughs in statistics: Methodology and distribu- tion, pp. 66-70. Springer, 1970.
  11. Gendron, G., Bao, Q., Witbrock, M., and Dobbie, G. Large language models are not strong abstract reasoners, 2024. URL https://arxiv.org/abs/2305.19555. · arXiv
  12. Godfrey-Smith, P. Theory and reality: An introduction to the philosophy of science. University of Chicago Press, 2009.
  13. Goodman, N. Fact, Fiction, and Forecast. Harvard Univer- sity Press, Cambridge, MA, 1983.
  14. Grünwald, P., de Heide, R., and Koolen, W. M. Safe testing. In 2020 Information Theory and Applications Workshop (ITA), pp. 1-54. IEEE, 2020.
  15. Gu, K., Shang, R., Jiang, R., Kuang, K., Lin, R.-J., Lyu, D., Mao, Y., Pan, Y., Wu, T., Yu, J., Zhang, Y., Zhang, T. M., Zhu, L., Merrill, M. A., Heer, J., and Althoff, T. Blade: Benchmarking language model agents for data- driven science, 2024. URL https://arxiv.org/ abs/2408.09667. · arXiv
  16. Han, S. J., Ransom, K., Perfors, A., and Kemp, C. Inductive reasoning in humans and large language models, 2023. URL https://arxiv.org/abs/2306.06548. · arXiv
  17. Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to nat- ural language task descriptions. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1935- 1952, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi: 10.18653/v1/2023.acl-long. 108. URL https://aclanthology.org/2023. acl-long.108. · DOI
  18. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. A survey on hallucination in large language models: Principles, taxon- omy, challenges, and open questions. ACM Transactions on Information Systems, 2023.
  19. Ifargan, T., Hafner, L., Kern, M., Alcalay, O., and Kishony, R. Autonomous llm-driven research from data to human- verifiable research papers, 2024. URL https:// arxiv.org/abs/2404.17605. · arXiv
  20. Ioannidis, J. P. Why most published research findings are false. PLoS medicine, 2(8):e124, 2005.
  21. Jun, E., Birchfield, M., De Moura, N., Heer, J., and Just, R. Hypothesis formalization: Empirical findings, software limitations, and design implications. ACM Transactions on Computer-Human Interaction (TOCHI), 29(1):1-28, 2022.
  22. Kuhn, T. S. The Structure of Scientific Revolutions. Univer- sity of Chicago Press, Chicago, 1st edition, 1962.
  23. Lakatos, I. The Methodology of Scientific Research Pro- grammes. Cambridge University Press, Cambridge, 1978.
  24. Lehr, S. A., Caliskan, A., Liyanage, S., and Banaji, M. R. Chatgpt as research scientist: Probing gpt's capabilities as a research librarian, research ethicist, data generator and data predictor, 2024. URL https://arxiv.org/ abs/2406.14765. · arXiv
  25. Li, M. Y., Vajipey, V., Goodman, N. D., and Fox, E. B. Critical: Critic automation with language models. arXiv preprint arXiv:2411.06590, 2024a. · arXiv
  26. Li, R., Patel, T., Wang, Q., and Du, X. Mlr-copilot: Au- tonomous machine learning research based on large lan- guage models agents, 2024b. URL https://arxiv. org/abs/2408.14033.
  27. Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., McFarland, D., and Zou, J. Can large language models provide useful feedback on research papers? a large-scale empirical analysis, 2023. URL https://arxiv.org/abs/ 2310.01783.
  28. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https: //arxiv.org/abs/2408.06292.
  29. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. NeurIPS, 36, 2024.
  30. Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725, 2024. · arXiv
  31. Manning, B. S., Zhu, K., and Horton, J. J. Automated social science: Language models as scientist and subjects, 2024. URL https://arxiv.org/abs/2404.11794. · arXiv
  32. Maxwell, N. Popper, kuhn, lakatos and aim-oriented em- piricism. arXiv preprint, 2012.
  33. Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines, 2023. URL https://arxiv.org/abs/2307.04721. · arXiv
  34. Moskvichev, A., Odouard, V. V., and Mitchell, M. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain, 2023. URL https: //arxiv.org/abs/2305.07141.
  35. Neyman, J. and Pearson, E. S. On the use and interpretation of certain test criteria for purposes of statistical inference part i. Biometrika, 20(1-2):175-240, 1928.
  36. Neyman, J. and Pearson, E. S. The testing of statistical hypotheses in relation to probabilities a priori. In Math- ematical proceedings of the Cambridge philosophical society, volume 29, pp. 492-510. Cambridge University Press, 1933.
  37. Popper, K. The logic of scientific discovery. Routledge, 2005.
  38. Press, C. U. Normal science and dogmatism, paradigms and progress: Kuhn versus popper and lakatos. 2009.
  39. Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., and Bethge, M. Citeme: Can language models accurately cite scientific claims?, 2024. URL https://arxiv.org/abs/2407.12861. · arXiv
  40. Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhaga- vatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., and Ren, X. Phenomenal yet puzzling: Testing inductive rea- soning capabilities of language models with hypothesis refinement, 2024. URL https://arxiv.org/abs/ 2310.08559.
  41. Ridnik, T., Kredo, D., and Friedman, I. Code generation with alphacodium: From prompt engineering to flow en- gineering. arXiv preprint arXiv:2401.08500, 2024. · arXiv
  42. Rubin, M. The replication crisis is less of a "crisis" in lakatos' philosophy of science. European Journal for Philosophy of Science, 15(5), 2025. doi: 10.1007/ s13194-024-00629-x. · DOI
  43. Shafer, G. The language of betting as a strategy for sta- tistical and scientific communication. arXiv preprint arXiv:1903.06991, 2019. · arXiv
  44. Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv. org/abs/2409.04109.
  45. Tang, X., Zheng, Z., Li, J., Meng, F., Zhu, S.-C., Liang, Y., and Zhang, M. Large language models are in-context semantic reasoners rather than symbolic reasoners, 2023. URL https://arxiv.org/abs/2305.14825. · arXiv
  46. Thompson, W. H. and Skau, S. On the scope of scientific hypotheses. Royal Society Open Science, 10(8):230607, 2023.
  47. Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., Ma, Y., Tong, H., Trinh, K., Tian, C., Wang, Z., Wu, B., Xiong, Y., Yin, S., Zhu, M., Lieret, K., Lu, Y., Liu, G., Du, Y., Tao, T., Press, O., Callan, J., Huerta, E., and Peng, H. Scicode: A research coding benchmark curated by scientists, 2024. URL https://arxiv.org/abs/ 2407.13168.
  48. van Fraassen, B. C. The Scientific Image. Clarendon Press, Oxford, 1980.
  49. Vovk, V. and Wang, R. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736- 1754, 2021.
  50. Wang, Q., Downey, D., Ji, H., and Hope, T. Scimon: Scien- tific inspiration machines optimized for novelty, 2024a. URL https://arxiv.org/abs/2305.14259. · arXiv
  51. Wang, R. and Ramdas, A. False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):822-852, 2022.
  52. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models. ICLR, 2024b.
  53. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models, 2024c. URL https://arxiv. org/abs/2309.05660.
  54. Webb, T., Holyoak, K. J., and Lu, H. Emergent analogical reasoning in large language models, 2023. URL https: //arxiv.org/abs/2212.09196.
  55. Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., and Cambria, E. Are large language models really good logical reasoners? a comprehensive evaluation and beyond, 2024a. URL https://arxiv.org/abs/2306.09841. · arXiv
  56. Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. Llms and the abstraction and reasoning corpus: Suc- cesses, failures, and the importance of object-based rep- resentations, 2024b. URL https://arxiv.org/ abs/2305.18354. · arXiv
  57. Yang, Z., Dong, L., Du, X., Cheng, H., Cambria, E., Liu, X., Gao, J., and Wei, F. Language models as inductive reasoners, 2024a. URL https://arxiv.org/abs/ 2212.10923.
  58. Yang, Z., Du, X., Li, J., Zheng, J., Poria, S., and Cambria, E. Large language models for automated open-domain scientific hypotheses discovery, 2024b. URL https: //arxiv.org/abs/2309.02726.
  59. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. ICLR, 2023.
  60. Zhang, X., Xie, Y., Huang, J., Ma, J., Pan, Z., Liu, Q., Xiong, Z., Ergen, T., Shim, D., Lee, H., and Mei, Q. Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows, 2024. URL https://arxiv.org/abs/ 2406.06357.
  61. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
  62. Zhou, Y., Liu, H., Srivastava, T., Mei, H., and Tan, C. Hy- pothesis generation with large language models. arXiv preprint arXiv:2404.04326, 2024. Critic: 1. Causality Direction: If the main hypothesis (LCP2 regulates IL2) is null, this test might still show significance if there's a third factor affecting both genes, or if IL2 regulates LCP2. Therefore, the test doesn't fully satisfy the crucial property that null main hypothesis implies null sub-hypothesis. · arXiv

Authors
Huang, K., Jin, Y., Li, R., Li, M. Y., Candès, E., & Leskovec, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.09858
arXiv
2502.09858
OpenAlex
W4407632372
Cited by
7
References
0
Citations by year
2026: 5 · 2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1835598, 2403318, 1918940)
  • Genentech
  • Wu Tsai Neurosciences Institute, Stanford University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
hypothesis verification, falsification, Karl Popper, LLM agents, large language models, hypothesis generation, hallucinations, sequential testing, type I error control, statistical power

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review