CoChat
Sign Up
PreprintCited by 1Topic ModelingArtificial Intelligence ApplicationsAI-based Problem Solving and Planning

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

Open access
Abramovich, Talor, Chechik, Gal·2025·arXiv
DOI 10.48550/arxiv.2507.08038arXiv 2507.08038
Open this paper in CoChat — free accountOpen PDF

Abstract

Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge. A key mechanism to obtain such insights is through ablation experiments. To this end, we introduce AblationBench, a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section and contains 83 instances, and ReviewerAblation, which helps reviewers find missing ablations in a full paper and contains 350 instances. For both tasks, we develop LM-based judges that serve as an automatic evaluation framework. Our experiments with frontier LMs show that these tasks remain challenging, with the best-performing LM system identifying only 45% of the original ablations on average, below human-level performance. We observe an inverse performance trend between the author and reviewer tasks, which we attribute to differences in model grounding. Lastly, we analyze the limitations of current LMs on these tasks, and find that chain-of-thought prompting outperforms an agent-based approach. Our data is available on https://huggingface.co/collections/ai-coscientist/ablationbench, and our code is available on https://github.com/ai-scientist-bench/ablation-bench .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.08038

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work has focused primarily on code implementation and execution, overlooking experimental design, and no benchmark exists to evaluate AI's ability to conduct systematic experiment design.

    “While experimental design is a critical stage of the research process, prior work has focused primarily on code implementation and execution, overlooking the importance of this stage, and no benchmark exists to evaluate AI's ability to conduct systematic experiment design.”

    From Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design (2026) · cites this paper as Abramovich and Chechik 2025

    The Related Work section supports the observation that many earlier benchmarks focused on experiment replication and code generation. But it explicitly identifies AbGen as an existing benchmark for ablation planning, contradicting the claim that no benchmark evaluated AI experiment design; the paper's own AblationBench also evaluates that capability (Abstract). No page numbers are available in our copy. (LLM, conf=0.87)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Existing benchmarks related to experimentation either focus exclusively on code execution or remain confined to isolated sub-tasks such as ablation generation, falling short of systematic end-to-end assessment.

    “Existing benchmarks (Kon et al. 2025;Abramovich and Chechik 2025;Zhao et al. 2025) related to experimentation either focus exclusively on code execution or, in the few cases that address experimental design, remain confined to isolated sub-tasks such as ablation generation, falling short of systematic end-to-end assessment.”

    From Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design (2026) · cites this paper as Abramovich and Chechik 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 70

  1. H. Abdullah. Boosting membership inference attacks with upstream modification, 2025. URL https:// openreview.net/forum?id=nAK26c8s9X.
  2. T. Ahmed, P. Devanbu, C. Treude, and M. Pradel. Can llms replace manual annotation of software engineering artifacts?, 2025. URL https://arxiv.org/abs/ 2408.05534.
  3. A. Ajith, M. Xia, A. Chevalier, T. Goyal, D. Chen, and T. Gao. Litsearch: A retrieval benchmark for scientific literature search, 2024. URL https://arxiv.org/ abs/2407.18940. · arXiv
  4. A. Audran-Reiss, J. Armengol-Estapé, K. Hambardzumyan, A. Budhiraja, M. Josifoski, E. Toledo, R. Hazra, D. Magka, M. Shvartsman, P. Pathak, J. T. Kao, L. Cipolina-Kun, B. Gauri, J.-C. Gagnon-Audet, E. Tewolde, J. Zhang, T. Cohen, Y. Adi, T. Shavrina, and Y. Bachrach. What does it take to be a good ai re- search agent? studying the role of ideation diversity, 2025. URL https://arxiv.org/abs/2511.15593. · arXiv
  5. Y. Bang, Z. Ji, A. Schelten, A.-. thony Hartshorn, T. Fowler, C. Zhang, N. Cancedda, and P. Fung. Hallulens: Llm hallucination benchmark. ArXiv, abs/2504.17550, 2025. URL https://api.semanticscholar. org/CorpusID:278033255. · arXiv
  6. B. Bogin, K. Yang, S. Gupta, K. Richardson, E. Bran- som, P. Clark, A. Sabharwal, and T. Khot. Super: Eval- uating agents on setting up and executing tasks from research repositories, 2024. URL https://arxiv. org/abs/2409.07440.
  7. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, 2020. URL https: //arxiv.org/abs/2005.14165.
  8. S. Bubeck, C. Coester, R. Eldan, T. Gowers, Y. T. Lee, A. Lupsasca, M. Sawhney, R. Scherrer, M. Sellke, B. K. Spears, D. Unutmaz, K. Weil, S. Yin, and N. Zhivotovskiy. Early science acceleration experiments with gpt-5, 2025. URL https://arxiv.org/abs/2511.16072. · arXiv
  9. C.-M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu. Chateval: Towards better llm-based evalua- tors through multi-agent debate. ArXiv, abs/2308.07201, 2023. URL https://api.semanticscholar. org/CorpusID:260887105. · arXiv
  10. J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. M ądry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025. URL https://arxiv.org/abs/2410.07095. · arXiv
  11. D. Chen, Z. Li, M. Fan, C. Chen, W. Zhou, Y. Wang, and Y. Li. Growth inhibitors for suppressing inappropriate image concepts in diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=w4C4z80w59.
  12. X. Chen, N. Zhang, L. Li, Y. Yao, S. Deng, C. Tan, F. Huang, L. Si, and H. Chen. Good visual guidance make a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction. In M. Carpuat, M.-C. de Marn- effe, and I. V. Meza Ruiz, editors, Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 1607-1618, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. findings-naacl.121. URL https://aclanthology. org/2022.findings-naacl.121/. · DOI
  13. C.-H. Chiang and H.-y. Lee. Can large language models be an alternative to human evaluations? In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 15607-15631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-long.870. URL https://aclanthology.org/ 2023.acl-long.870/. · DOI · Paper page
  14. V. P. Dwivedi and X. Bresson. A generalization of trans- former networks to graphs, 2021. URL https:// arxiv.org/abs/2012.09699. · arXiv
  15. J. Fu, S.-K. Ng, Z. Jiang, and P. Liu. Gptscore: Evaluate as you desire. In North American Chap- ter of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar. org/CorpusID:256662188.
  16. L. Gessler, S. Behzad, Y. J. Liu, S. Peng, Y. Zhu, and A. Zeldes. DisCoDisCo at the DISRPT2021 shared task: A system for discourse segmentation, classifica- tion, and connective detection. In A. Zeldes, Y. J. Liu, M. Iruskieta, P. Muller, C. Braud, and S. Badene, edi- tors, Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021), pages 51-62, Punta Cana, Dominican Republic, Nov. 2021. As- sociation for Computational Linguistics. doi: 10.18653/ v1/2021.disrpt-1.6. URL https://aclanthology. org/2021.disrpt-1.6/. Google. Start building with gemini 2.5 flash. https://developers.googleblog.com/en/ start-building-with-gemini-25-flash/, 2025. · DOI
  17. Google Research. arXiv LaTeX Cleaner, 2025. URL https://github.com/google-research/ arxiv-latex-cleaner.
  18. J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, K. Saab, D. Popovici, J. Blum, F. Zhang, K. Chou, A. Hassidim, B. Gokturk, A. Vahdat, P. Kohli, Y. Matias, A. Carroll, K. Kulkarni, N. Tomasev, Y. Guan, V. Dhillon, E. D. Vaishnav, B. Lee, T. R. D. Costa, J. R. Pe- nadés, G. Peltz, Y. Xu, A. Pawlosky, A. Karthikesalingam, and V. Natarajan. Towards an ai co-scientist, 2025. URL https://arxiv.org/abs/2502.18864. · arXiv
  19. A. Grosnit, A. Maraval, J. Doran, G. Paolo, A. Thomas, R. S. H. N. Beevi, J. Gonzalez, K. Khandelwal, I. Iacobacci, A. Benechehab, H. Cherkaoui, Y. A. El-Hili, K. Shao, J. Hao, J. Yao, B. Kegl, H. Bou-Ammar, and J. Wang. Large language models orchestrating structured reasoning achieve kaggle grandmaster level, 2024. URL https: //arxiv.org/abs/2411.03562.
  20. K. Gu, R. Shang, R. Jiang, K. Kuang, R.-J. Lin, D. Lyu, Y. Mao, Y. Pan, T. Wu, J. Yu, Y. Zhang, T. M. Zhang, L. Zhu, M. A. Merrill, J. Heer, and T. Althoff. Blade: Benchmarking language model agents for data-driven science, 2024. URL https://arxiv.org/abs/ 2408.09667.
  21. Z. Hu, Y. Dong, K. Wang, K.-W. Chang, and Y. Sun. Gpt-gnn: Generative pre-training of graph neural net- works, 2020. URL https://arxiv.org/abs/ 2006.15437.
  22. Q. Huang, J. Vora, P. Liang, and J. Leskovec. Mlagentbench: Evaluating language agents on machine learning experi- mentation, 2024a. URL https://arxiv.org/abs/ 2310.03302.
  23. X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, R. Tang, and E. Chen. Under- standing the planning of llm agents: A survey. CoRR, abs/2402.02716, 2024b. URL https://doi.org/ 10.48550/arXiv.2402.02716. · DOI · Paper page
  24. T. Ifargan, L. Hafner, M. Kern, O. Alcalay, and R. Kishony. Autonomous llm-driven research from data to human- verifiable research papers, 2024. URL https:// arxiv.org/abs/2404.17605. · arXiv
  25. A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Ol- szewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. Wang, M. Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, Z. Zhang, S. Goldshtein, and D. Das. The facts grounding leader- board: Benchmarking llms' ability to ground responses to long-form input, 2025. URL https://arxiv.org/ abs/2501.03200. · arXiv
  26. Y. Jafarian and H. S. Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12753-12762, June 2021.
  27. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422-446, Oct. 2002. ISSN 1046-8188. doi: 10.1145/582415.582418. URL https://doi.org/ 10.1145/582415.582418. · DOI · Paper page
  28. Z. Li, Q. Zang, D. Ma, J. Guo, T. Zheng, M. Liu, X. Niu, Y. Wang, J. Yang, J. Liu, W. Zhong, W. Zhou, W. Huang, and G. Zhang. Autokaggle: A multi-agent framework for autonomous data science competitions, 2024. URL https://arxiv.org/abs/2410.20424. · arXiv
  29. S. Lin, A. Ryabtsev, S. Sengupta, B. Curless, S. Seitz, and I. Kemelmacher-Shlizerman. Real-time high-resolution background matting, 2020. URL https://arxiv. org/abs/2012.07810.
  30. Z. C. Lipton and J. Steinhardt. Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research. Queue, 17(1):45-77, Feb. 2019. ISSN 1542-7730. doi: 10. 1145/3317287.3328534. URL https://doi.org/ 10.1145/3317287.3328534. · DOI · Paper page
  31. R. Meyes, M. Lu, C. W. de Puiseau, and T. Meisen. Ablation studies in artificial neural networks, 2019. URL https: //arxiv.org/abs/1901.08644. MiniMax. Minimax m2.5: Built for real-world pro- ductivity. https://www.minimax.io/news/ minimax-m25, 2026.
  32. A. H. Nasution and A. Onan. Chatgpt label: Comparing the quality of human-generated and llm-generated anno- tations in low-resource language nlp tasks. IEEE Access, 12:71876-71900, 2024. doi: 10.1109/ACCESS.2024. 3402809. · DOI
  33. D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, A. Budhiraja, D. Magka, V. Vorotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. Foerster, Y. Bachrach, W. Y. Wang, and R. Raileanu. Mlgym: A new framework and bench- mark for advancing ai research agents, 2025. URL https://arxiv.org/abs/2502.14499. · arXiv
  34. OpenAI, :, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Ap- plebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, B. Barak, A. Bennett, T. Bertao, N. Brett, E. Brevdo, G. Brockman, S. Bubeck, C. Chang, K. Chen, M. Chen, E. Cheung, A. Clark, D. Cook, M. Dukhan, C. Dvo- rak, K. Fives, V. Fomenko, T. Garipov, K. Georgiev, M. Glaese, T. Gogineni, A. Goucher, L. Gross, K. G. Guzman, J. Hallman, J. Hehir, J. Heidecke, A. Helyar, H. Hu, R. Huet, J. Huh, S. Jain, Z. Johnson, C. Koch, I. Kofman, D. Kundel, J. Kwon, V. Kyrylov, E. Y. Le, G. Leclerc, J. P. Lennon, S. Lessans, M. Lezcano-Casado, Y. Li, Z. Li, J. Lin, J. Liss, Lily, Liu, J. Liu, K. Lu, C. Lu, Z. Martinovic, L. McCallum, J. McGrath, S. McK- inney, A. McLaughlin, S. Mei, S. Mostovoy, T. Mu, G. Myles, A. Neitz, A. Nichol, J. Pachocki, A. Paino, D. Palmie, A. Pantuliano, G. Parascandolo, J. Park, L. Pathak, C. Paz, L. Peran, D. Pimenov, M. Pokrass, E. Proehl, H. Qiu, G. Raila, F. Raso, H. Ren, K. Richard- son, D. Robinson, B. Rotsted, H. Salman, S. Sanjeev, M. Schwarzer, D. Sculley, H. Sikchi, K. Simon, K. Sing- hal, Y. Song, D. Stuckey, Z. Sun, P. Tillet, S. Toizer, F. Tsimpourlas, N. Vyas, E. Wallace, X. Wang, M. Wang, O. Watkins, K. Weil, A. Wendling, K. Whinnery, C. Whit- ney, H. Wong, L. Yang, Y. Yang, M. Yasunaga, K. Ying, W. Zaremba, W. Zhan, C. Zhang, B. Zhang, E. Zhang, and S. Zhao. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925. · arXiv
  35. OpenReview. OpenReview.net, 2025. URL https:// openreview.net/.
  36. O. Press, A. Hochlehnert, A. Prabhu, V. Udandarao, O. Press, and M. Bethge. CiteME: Can language models accurately cite scientific claims? In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: //openreview.net/forum?id=S9Qrrxpy6z.
  37. P. Roetzer and F. Bernard. Spidermatch: 3d shape match- ing with global optimality and geometric consistency. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024.
  38. M. Seo, J. Baek, S. Lee, and S. J. Hwang. Paper2code: Automating code generation from scientific papers in machine learning, 2025. URL https://arxiv.org/ abs/2504.17192. · arXiv
  39. Y. Shao, Y. Jiang, T. A. Kanell, P. Xu, O. Khattab, and M. S. Lam. Assisting in writing wikipedia-like articles from scratch with large language models, 2024. URL https://arxiv.org/abs/2402.14207. · arXiv
  40. J. Shi, G. Gare, J. Tian, S. Chai, Z. Lin, A. Vasudevan, D. Feng, F. Ferroni, and S. Kong. Lca-on-the-line: Bench- marking out-of-distribution generalization with class tax- onomies, 2024. URL https://arxiv.org/abs/ 2407.16067.
  41. A. Shypula, A. Madaan, Y. Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, Y. Jin, Y. Jiang, Y. Wang, Y. Wang, Y. Liu, Z. Stuben- voll, Z. Dou, Z. Wu, and Z. Wang. Openai gpt-5 sys- tem card, 2025. URL https://arxiv.org/abs/ 2601.03267.
  42. M. D. Skarlinski, S. Cox, J. M. Laurent, J. D. Braza, M. Hinks, M. J. Hammerling, M. Ponnapati, S. G. Ro- driques, and A. D. White. Language agents achieve su- perhuman synthesis of scientific knowledge, 2024. URL https://arxiv.org/abs/2409.13740. · arXiv
  43. Z. Song, J. Lu, Y. Du, B. Yu, T. M. Pruyn, Y. Huang, K. Guo, X. Luo, Y. Qu, Y. Qu, Y. Wang, H. Wang, J. Guo, J. Gan, P. Shojaee, D. Luo, A. M. Bran, G. Li, Q. Zhao, S.- X. L. Luo, Y. Zhang, X. Zou, W. Zhao, Y. F. Zhang, W. Zhang, S. Zheng, S. Zhang, S. T. Khan, M. Rajabi- Kochi, S. Paradi-Maropakis, T. Baltoiu, F. Xie, T. Chen, K. Huang, W. Luo, M. Fang, X. Yang, L. Cheng, J. He, S. Hassoun, X. Zhang, W. Wang, C. K. Reddy, C. Zhang, Z. Zheng, M. Wang, L. Cong, C. P. Gomes, C.-Y. Hsieh, A. Nandy, P. Schwaller, H. J. Kulik, H. Jia, H. Sun, S. M. Moosavi, and C. Duan. Evaluating large language models in scientific discovery, 2025. URL https://arxiv. org/abs/2512.15567.
  44. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan. Paperbench: Evaluating ai's ability to replicate ai research, 2025. URL https://arxiv.org/abs/2504.01848. · arXiv
  45. V. Surkov, C. Wendler, M. Terekhov, J. Deschenaux, R. West, and C. Gulcehre. Unpacking SDXL turbo: Inter- preting text-to-image models with sparse autoencoders, 2024. URL https://openreview.net/forum? id=Ch8s4FdUXS.
  46. J. Tang, L. Xia, Z. Li, and C. Huang. Ai-researcher: Au- tonomous scientific innovation, 2025. URL https: //arxiv.org/abs/2505.18705.
  47. Q. Team. Qwen3.5-omni technical report, 2026. URL https://arxiv.org/abs/2604.15804. · arXiv
  48. M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, S. Liu, D. Luo, Y. Ma, H. Tong, K. Trinh, C. Tian, Z. Wang, B. Wu, Y. Xiong, S. Yin, M. Zhu, K. Lieret, Y. Lu, G. Liu, Y. Du, T. Tao, O. Press, J. Callan, E. Huerta, and H. Peng. Scicode: A research coding benchmark curated by scientists, 2024a. URL https://arxiv.org/abs/2407.13168. · arXiv
  49. Y. Tian, H. Chen, C. Xu, and Y. Wang. Image processing gnn: Breaking rigidity in super-resolution. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24108-24117, June 2024b.
  50. P. Verga, S. Hofstatter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis. Replacing judges with juries: Evaluating llm gener- ations with a panel of diverse models, 2024. URL https://arxiv.org/abs/2404.18796. · arXiv
  51. L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.- W. Lee, and E.-P. Lim. Plan-and-solve prompt- ing: Improving zero-shot chain-of-thought reason- ing by large language models. In Annual Meet- ing of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar. org/CorpusID:258558102.
  52. P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. In L.-W. Ku, A. Mar- tins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440-9450, Bangkok, Thailand, Aug. 2024. Association for Com- putational Linguistics. doi: 10.18653/v1/2024.acl-long. 511. URL https://aclanthology.org/2024. acl-long.511/. · DOI · Paper page
  53. J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, edi- tors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=_VjQlMeSB_J.
  54. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language mod- els, 2023. URL https://arxiv.org/abs/2201. 11903.
  55. H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts, 2024. URL https://arxiv.org/abs/ 2411.15114.
  56. Y. Xiao, R. Wang, L. Kong, D. Golac, and W. Wang. Csr-bench: Benchmarking llm agents in deployment of computer science research repositories, 2025. URL https://arxiv.org/abs/2502.06111. · arXiv
  57. W. Xu, G. Zhu, X. Zhao, L. Pan, L. Li, and W. Wang. Pride and prejudice: LLM amplifies self-bias in self- refinement. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474-15492, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.826. URL https:// aclanthology.org/2024.acl-long.826/. · DOI · Paper page
  58. H. Yang, S. Zhang, D. Huang, X. Wu, H. Zhu, T. He, S. TANG, H. Zhao, Q. Qiu, B. Lin, X. He, and W. Ouyang. UniPAD: A universal pre-training paradigm for autonomous driving, 2024a. URL https:// openreview.net/forum?id=i92ssjkZCz.
  59. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://arxiv. org/abs/2405.15793.
  60. Y. Yang, K. Zha, Y.-C. Chen, H. Wang, and D. Katabi. Delving into deep imbalanced regression, 2021. URL https://arxiv.org/abs/2102.09554. · arXiv
  61. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. R. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://openreview.net/forum? id=5Xc1ecxO1h.
  62. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, 2023b. URL https: //openreview.net/forum?id=WE_vluYUL-X.
  63. A. Zeng, L. Yang, X. Ju, J. Li, J. Wang, and Q. Xu. Smooth- net: A plug-and-play network for refining human poses in videos, 2022. URL https://arxiv.org/abs/ 2112.13715.
  64. N. Zhang, L. Li, X. Chen, S. Deng, Z. Bi, C. Tan, F. Huang, and H. Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=ek9a0qIafW.
  65. Z. Zhang, A. Zhu, L. Yang, Y. Xu, L. Li, P. M. Phothilimthana, and Z. Jia. Accelerating retrieval- augmented language model serving with speculation, 2024. URL https://openreview.net/forum? id=vkzPuZJ80a.
  66. Y. Zhao, W. Chen, Z. Xu, M. Patwardhan, C. Wang, Y. Liu, L. Vig, and A. Cohan. AbGen: Evaluating large language models in ablation study design and eval- uation for scientific research. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 12479-12491, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251- 0. doi: 10.18653/v1/2025.acl-long.611. URL https: //aclanthology.org/2025.acl-long.611/. · DOI
  67. J. Zheng, Q. Ma, S. Qiu, Y. Wu, P. Ma, J. Liu, H. Feng, X. Shang, and H. Chen. Preserving commonsense knowl- edge from pre-trained language models via causal in- ference, 2023a. URL https://arxiv.org/abs/ 2306.10790.
  68. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b. URL https: //openreview.net/forum?id=uccHPGDlao.
  69. H. Zhou, X. Wan, L. Proleev, D. Mincu, J. Chen, K. Heller, and S. Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering, 2024. URL https://arxiv.org/abs/2309.17249. · arXiv
  70. M. Zhuge, C. Zhao, D. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, Y. Shi, V. Chandra, and J. Schmidhuber. Agent-as-a- judge: Evaluate agents with agents, 2024. URL https: //arxiv.org/abs/2410.10934.

Cited by

Sorted by citations

Authors
Abramovich, T., & Chechik, G.
ORCID
Chechik, Gal
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
30612-30651
DOI
10.48550/arxiv.2507.08038
arXiv
2507.08038
OpenAlex
W6967013885
Cited by
0
References
0
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
ablation study, automated planning, language model agents, benchmark suite, scientific writing, peer review, automated assessment, chain-of-thought prompting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review