CoChat
Sign Up
PreprintCited by 1Topic ModelingDomain Adaptation and Few-Shot LearningArtificial Intelligence Applications

How to Train Your Advisor: Steering Black-Box LLMs with Advisor Models

Open access
Asawa, Parth, Zhu, Alan, O'Neill, Abigail, Zaharia, Matei, Dimakis, Alexandros G., Gonzalez, Joseph E.·2025·arXiv
DOI 10.48550/arxiv.2510.02453arXiv 2510.02453
Open this paper in CoChat — free accountOpen PDF

Abstract

Frontier language models are deployed as black-box services, where model weights cannot be modified and customization is limited to prompting. We introduce Advisor Models, a method to train small open-weight models to generate dynamic, per-instance natural language advice that improves the capabilities of black-box frontier models. Advisor Models improve GPT-5.2's performance on RuleArena (Taxes) by 27.4%, reduce Gemini 3 Pro's steps taken in SWE agent tasks by 24.6%, and outperform static prompt optimizers in personalizing GPT-5 to user preferences (85-100% vs. 40-60%). We also find that advisors are transferable: an advisor trained with a low-cost student model still transfers improvements to a frontier model. Moreover, Advisor Models are robust: we observe no degradation on other benchmarks than the pipeline is trained on. Our method shows how to perform parametric optimization for black-box frontier models in a practical and cost-effective way.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.02453

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The memory agent is more constrained than advisor-style methods: it is not intended to provide broad strategic advice or perform general deliberation.

    “Our memory agent is related to these supervisory approaches, but is more constrained: it is not intended to provide broad strategic advice or perform general deliberation.”

    From Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents (2026) · cites this paper as Asawa et al., 2025

  • Advisor-style systems use an auxiliary model that observes the trajectory and provides natural language guidance to an executor.

    “This is analogous to advisor-style systems, where an auxiliary model observes the trajectory and provides naturallanguage guidance to an executor (Asawa et al., 2025;Anthropic, 2026).”

    From Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents (2026) · cites this paper as Asawa et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Agarwal, E., Magazine, R., Singh, J., Dani, V., Ganu, T., and Nambi, A. PromptWizard: Optimizing prompts via task-aware, feedback-driven self-evolution. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 19974-20003, 2025.
  2. Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., Potts, C., Sen, K., Dimakis, A., Stoica, I., Klein, D., Zaharia, M., and Khattab, O. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview .net/forum?id=RQm2KQTM5r. Anthropic. Introducing Claude Haiku 4.5. hhttps://ww w.anthropic.com/news/claude-haiku-4-5, October 2025a.
  3. Bae, S., Hong, J., Lee, M. Y., Kim, H., Nam, J., and Kwak, D. Online difficulty filtering for reasoning oriented re- inforcement learning. In Proceedings of the 19th Con- ference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 700-719, 2026.
  4. Cheng, J., Liu, X., Zheng, K., Ke, P., Wang, H., Dong, Y., Tang, J., and Huang, M. Black-box prompt optimization: Aligning large language models without model training. In Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 3201-3219, 2024.
  5. Cuadron, A., Li, D., Ma, W., Wang, X., Wang, Y., Zhuang, S., Liu, S., Schroeder, L. G., Xia, T., Mao, H., et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025. · arXiv
  6. Fernando, C., Banarse, D. S., Michalewski, H., Osindero, S., and Rocktäschel, T. PromptBreeder: Self-referential self- improvement via prompt evolution. In International Con- ference on Machine Learning, pp. 13481-13544.
  7. PMLR, 2024.
  8. Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 1(5):6, 2025a. Gemini Team. Gemini 3 Pro: the frontier of vision AI. https://blog.google/innovation-and-a i/technology/developers-tools/gemin i-3-pro-vision/, December 2025b. · arXiv
  9. Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Le Roux, N. VinePPO: Refining credit assignment in RL training of LLMs. In In- ternational Conference on Machine Learning, pp. 29557- 29590. PMLR, 2025.
  10. Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge- intensive NLP. arXiv preprint arXiv:2212.14024, 2022. URL https://arxiv.org/abs/2212.14024. · arXiv
  11. Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., A, S. V., Haq, S., Sharma, A., Joshi, T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy: Compiling declarative language model calls into state-of- the-art pipelines. In International Conference on Learn- ing Representations, volume 2024, pp. 54928-54958, 2024. URL https://proceedings.iclr.cc/ paper_files/paper/2024/file/f1cf02ce 09757f57c3b93c0db83181e0-Paper-Confe rence.pdf.
  12. Kong, W., Hombaiah, S., Zhang, M., Mei, Q., and Bender- sky, M. PRewrite: Prompt rewriting with reinforcement learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 594-601, 2024.
  13. Li, C., Zhuang, Y., Qiang, R., Sun, H., Dai, H., Zhang, C., and Dai, B. Matryoshka Pilot: Learning to drive black-box llms with LLMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
  14. Li, Z., Peng, B., He, P., Galley, M., Gao, J., and Yan, X. Guiding large language models via directional stimulus prompting. Advances in Neural Information Processing Systems, 36:62630-62656, 2023.
  15. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In International Conference on Learning Representations, volume 2024, pp. 39578-39601, 2024.
  16. Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 2025.
  17. Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P. Curriculum learning for reinforce- ment learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50, 2020.
  18. Opsahl-Ong, K., Ryan, M. J., Purtell, J., Broman, D., Potts, C., Zaharia, M., and Khattab, O. Optimizing in- structions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pp. 9340-9366, Miami, Florida, USA, nov 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.525. URL https://ac lanthology.org/2024.emnlp-main.525/. · DOI · Paper page
  19. Popović, M. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation, pp. 392-395, 2015.
  20. Qwen Team. Qwen3 technical report, 2025. URL https: //arxiv.org/abs/2505.09388.
  21. Richardson, C., Zhang, Y., Gillespie, K., Kar, S., Singh, A., Raeesy, Z., Khan, O. Z., and Sethy, A. Integrating summarization and retrieval for enhanced personalization via large language models. Personalized Generative AI Workshop (PGAI) at CIKM 2023, 2023.
  22. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. · arXiv
  23. Singh, A., Hsu, S., Hsu, K., Mitchell, E., Ermon, S., Hashimoto, T., Sharma, A., and Finn, C. FSPO: Few-shot optimization of synthetic preferences effectively personal- izes to real users. In The Fourteenth International Confer- ence on Learning Representations, 2026. URL https: //openreview.net/forum?id=SzEc5fSBXv.
  24. Tanzer, G., Suzgun, M., Visser, E., Jurafsky, D., and Melas- Kyriazi, L. A benchmark for learning to translate a new language from one grammar book. In International Con- ference on Learning Representations, volume 2024, pp. 18955-18985, 2024.
  25. Wu, Z., Wang, S., Gu, J., Hou, R., Dong, Y., Vydiswaran, V., and Ma, H. IDPG: An instance-dependent prompt generation method. In Carpuat, M., de Marneffe, M.- C., and Meza Ruiz, I. V. (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 5507-5521, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.403. URL https://ac lanthology.org/2022.naacl-main.403/. · DOI
  26. Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In International Conference on Learning Representations, volume 2024, pp. 12028-12068, 2024a. URL https: //proceedings.iclr.cc/paper_files/pa per/2024/file/3339f19c5fcee3ad745029 47a32be9e6-Paper-Conference.pdf.
  27. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K. R., and Press, O. SWE-agent: Agent- computer interfaces enable automated software engi- neering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://arxiv.org/abs/2405.15793. · arXiv
  28. Yang, J., Lieret, K., Jimenez, C., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., and Yang, D. SWE-Smith: Scaling data for software engineering agents. Advances in Neural Information Processing Sys- tems, 38, 2026.
  29. Zhang, Z., Zheng, C., Tang, D., Sun, K., Ma, Y., Bu, Y., Zhou, X., and Zhao, L. Balancing specialized and general skills in LLMs: The impact of modern tuning and data strategy. arXiv preprint arXiv:2310.04945, 2023. · arXiv
  30. Zhou, R., Hua, W., Pan, L., Cheng, S., Wu, X., Yu, E., and Wang, W. Y. RuleArena: A benchmark for rule-guided reasoning with LLMs in real-world scenarios. In Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 550-572, 2025.
  31. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human- level prompt engineers. In The Eleventh International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id= 92gvk82DE-.
  32. Zhu, B., Jordan, M., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. In International Conference on Machine Learning, pp. 62405-62428. PMLR, 2024.

Cited by

Sorted by citations

Authors
Asawa, P., Zhu, A., O'Neill, A., Zaharia, M., Dimakis, A. G., & Gonzalez, J. E.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.02453
arXiv
2510.02453
OpenAlex
W4416368241
Cited by
1
References
1
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2146752, 2019844, DGE 2146 752)
  • Accenture
  • Nvidia
  • Samsung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
open-weight models, parameter optimization, knowledge distillation, transfer learning, personalization, prompt optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review