CoChat
Sign Up
PreprintCited by 1Topic ModelingMathematics, Computing, and Information ProcessingSoftware Engineering Research

A Compute-Matched Re-Evaluation of TroVE on MATH

Open access
Sesterhenn, Tobias, Berlot-Attwell, Ian, Zenkner, Janis, Bartelt, Christian·2025·arXiv
DOI 10.48550/arxiv.2507.22069arXiv 2507.22069
Open this paper in CoChat — free accountOpen PDF

Abstract

Reusing established theorems and formulas is central to mathematical problem solving, serving as essential building blocks for tackling increasingly complex challenges. Recent work, TroVE, argues that code-generating Large Language Models (LLMs) can benefit similarly on the MATH benchmark by inducing and reusing higher-level toolboxes. By allocating computational budget across an ensemble of three modes -- directly generating code, creating tools, and reusing tools -- TroVE claims to outperform a PRIMITIVE baseline that only performs direct generation. However, recent analysis (Berlot-Attwell et al., 2024) casts doubt on these gains, noting that the tools created are often trivial or rarely reused, suggesting that improvements may stem from self-consistency or self-correction. In this work, we re-evaluate TroVE on MATH, analyze the impact of each of its modes, and show that its benefit does not come from these mechanisms, but simply from a higher computational budget spent for TroVE compared to PRIMITIVE. To this end, we also perform a small correction in the original implementation of TroVE's selection mechanism, boosting TroVE's performance on MATH by 3\% in accuracy. After matching for compute, the benefit of TroVE reduces to a marginal improvement of 1\%, suggesting that this toolbox approach does not provide a significant benefit on MATH.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.22069

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Compute-matched evaluations show that gains from tool-augmented LLMs are largely driven by increased inference-time compute rather than effective reuse of learned tools, with limited evidence of consistent tool reuse across instances.

    “Compute-matched evaluations show that their gains are largely driven by increased inferencetime compute rather than effective reuse of learned tools, with limited evidence of consistent tool reuse across instances [Sesterhenn et al., 2025].”

    From ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis (2026) · cites this paper as Sesterhenn et al., 2025

    Section 4.1.1 reports that TROVE’s advantage on MATH appears to come primarily from repeated sampling rather than toolbox learning. Section 5.3 reports evidence that learned libraries are often invoked only once, casting doubt on reuse. The claim extends these findings to tool-augmented LLMs generally, whereas this paper’s compute-matched evaluation focuses on TROVE, chiefly on MATH. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 27

  1. Berlot-Attwell, I., Rudzicz, F., and Si, X. Library learning doesn't: The curious case of the single-use" library". arXiv preprint arXiv:2410.20274, 2024. · arXiv
  2. Berlot-Attwell, I., Rudzicz, F., and Si, X. Llm library learning fails: A lego-prover case study. arXiv preprint arXiv:2504.03048, 2025. · arXiv
  3. Bowers, M., Olausson, T. X., Wong, L., Grand, G., Tenen- baum, J. B., Ellis, K., and Solar-Lezama, A. Top-down synthesis for library learning. Proceedings of the ACM on Programming Languages, 7(POPL):1182-1213, 2023.
  4. Cai, T., Wang, X., Ma, T., Chen, X., and Zhou, D. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023. · arXiv
  5. Cao, D., Kunkel, R., Nandi, C., Willsey, M., Tatlock, Z., and Polikarpova, N. Babble: Learning better abstractions with e-graphs and anti-unification. Proceedings of the ACM on Programming Languages, 7(POPL):396-424, 2023.
  6. Cheng, Z., Dong, H., Wang, Z., Jia, R., Guo, J., Gao, Y., Han, S., Lou, J.-G., and Zhang, D. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.06712, 2021. · arXiv
  7. Ellis, K., Wong, L., Nye, M., Sable-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A., and Tenen- baum, J. B. Dreamcoder: growing generalizable, inter- pretable knowledge with wake-sleep bayesian program learning. Philosophical Transactions of the Royal Society A, 381(2251):20220050, 2023.
  8. Grand, G., Wong, L., Bowers, M., Olausson, T. X., Liu, M., Tenenbaum, J. B., and Andreas, J. Lilo: Learning interpretable libraries by compressing and documenting code. arXiv preprint arXiv:2310.19791, 2023. · arXiv
  9. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
  10. Li, W.-D. and Ellis, K. Is programming by example solved by llms? arXiv preprint arXiv:2406.08316, 2024. · arXiv
  11. Lin, X., Cao, Q., Huang, Y., Yang, Z., Liu, Z., Li, Z., and Liang, X. Atg: Benchmarking automated theorem gen- eration for generative language models. arXiv preprint arXiv:2405.06677, 2024. · arXiv
  12. Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Ra- jpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured math- ematical reasoning. arXiv preprint arXiv:2209.14610, 2022. · arXiv
  13. Pasupat, P. and Liang, P. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015. · arXiv
  14. Qian, C., Han, C., Fung, Y. R., Qin, Y., Liu, Z., and Ji, H. Creator: Tool creation for disentangling abstract and con- crete reasoning of large language models. arXiv preprint arXiv:2305.14318, 2023. · arXiv
  15. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
  16. Stengel-Eskin, E., Prasad, A., and Bansal, M. Regal: Refac- toring programs to discover generalizable abstractions. arXiv preprint arXiv:2401.16467, 2024. · arXiv
  17. Sweller, J. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2):257-285, 1988.
  18. Walther, C. and Kolbe, T. Proving theorems by reuse. Artifi- cial Intelligence, 116(1-2):17-66, 2000.
  19. Wang, H., Xin, H., Zheng, C., Li, L., Liu, Z., Cao, Q., Huang, Y., Xiong, J., Shi, H., Xie, E., et al. Lego-prover: Neural theorem proving with growing libraries. arXiv preprint arXiv:2310.00656, 2023. · arXiv
  20. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. · arXiv
  21. Wang, Z., Cheng, Z., Zhu, H., Fried, D., and Neubig, G. What are tools anyway? a survey from the language model perspective. arXiv preprint arXiv:2403.15452, 2024a. · arXiv
  22. Wang, Z., Fried, D., and Neubig, G. Trove: Inducing veri- fiable and efficient toolboxes for solving programmatic tasks. arXiv preprint arXiv:2401.12869, 2024b. · arXiv
  23. Wang, Z., Gandhi, A., Neubig, G., and Fried, D. Inducing programmatic skills for agentic tasks. arXiv e-prints, pp. arXiv-2504, 2025.
  24. Yuan, L., Chen, Y., Wang, X., Fung, Y. R., Peng, H., and Ji, H. Craft: Customizing llms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17428, 2023. · arXiv
  25. Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. · arXiv
  26. Zhao, B., Lucas, C. G., and Bramley, N. R. A model of conceptual bootstrapping in human cognition. Nature Human Behaviour, 8(1):125-136, 2024.
  27. Zheng, B., Fatemi, M. Y., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y., Gu, Y., Srinivasa, J., Liu, G., Neubig, G., et al. Skillweaver: Web agents can self-improve by discover- ing and honing skills. arXiv preprint arXiv:2504.07079, 2025. · arXiv

Cited by

Sorted by citations

Related

Authors
Sesterhenn, T., Berlot-Attwell, I., Zenkner, J., & Bartelt, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.22069
arXiv
2507.22069
OpenAlex
W4414930190
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
Trove, MATH benchmark, large language models, code generation, mathematical problem solving, tool reuse, computational budget, primitives, self-consistency, self-correction, selection mechanisms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review