Understanding the Challenges in Iterative Generative Optimization with LLMs
Open accessAbstract
Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Analysis of iterative generative optimization shows that design decisions (starting artifacts, credit horizons, experience batching) often dominate algorithm choice.
“Analysis of iterative generative optimization (Nie et al., 2026) shows that design decisions (starting artifacts, credit horizons, experience batching) often dominate algorithm choice”
From Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents (2026) · cites this paper as Nie et al., 2026
Cites
Bayesian Active Learning for Classification and Preference Learning
Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011) · arXiv · doi:10.48550/arxiv.1112.5745 · Cited by 507 · cited 1× · see citing works
“It is also closely related to active learning, where examples are selected to be put in a batch to make learning more efficient (Houlsby et al., 2011; Gal et al., 2017), as well as contrastive learning, where positive and negative examples are balanced in a batched setting (Chen et al., 2020;…”
Optimizing generative AI by backpropagating language model feedback
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Lu, P., Huang, Z., Guestrin, C., & Zou, J. (2025) · Nature · doi:10.1038/s41586-025-08661-4 · Cited by 85 · cited 4× · see citing works
“Agentic Libraries with Learning Loops Many frameworks enable LLMs to iteratively modify systems (Yang et al., 2024), particularly for prompt optimization (Khat-tab et al., 2024; Cheng et al., 2024; Yuksekgonul et al., 2025; Wang et al., 2024).”
InfluentialMemento: Fine-tuning LLM Agents without Fine-tuning LLMs
Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., Lee, K. Y., Zhang, G., Shao, K., Yang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.16153 · Cited by 2 · see citing works
MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
Zhou, Z., Qu, A., Wu, Z., Kim, S., Prakash, A., Rus, D., Zhao, J., Low, B. K. H., & Liang, P. P. (2025) · arXiv · doi:10.48550/arxiv.2506.15841 · cited 1×
“We note that there are many other forms of learning, such as building an explicit memory of past experiences (i.e., a generate-retrieve-summarize system) (Zhou et al., 2025b; Ouyang et al., 2025b), or updating LLM parameters during test time (i.e., test-time training) (Yuksekgonul et al., 2026).”
ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
Lange, R. T., Imajuku, Y., & Cetin, E. (2025) · arXiv · doi:10.48550/arxiv.2509.19349 · cited 1×
“We note that the learning loop can be implemented with any LLM optimization and search algorithms (Novikov et al., 2025; Pan et al., 2025a; Lange et al., 2025a; Agrawal et al., 2025; Ren et al., 2026) and the main focus of our paper is to study the factors that impact the learning loop, not the…”
Flagged first, then by citations
Cited by
Sorted by citations
Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents
Zhang, X., Wang, G., Cui, Y., Qiu, W., Li, Z., Zhu, B., & He, P. (2026) · arXiv
“Analysis of iterative generative optimization (Nie et al., 2026) shows that design decisions—starting artifacts, credit horizons, experience batching—often dominate algorithm choice, suggesting that fixing the meta-strategy while optimizing components may be more practical.”
- Authors
- Nie, A., Daull, X., Kuang, Z., Akkiraju, A., Chaudhuri, A., Piasevoli, M., Rong, R., Yuan, Y., Choudhary, P., Xiao, S., Fakoor, R., Swaminathan, A., & Cheng, C.-A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2603.23994
- OpenAlex
- W7140622660
- Cited by
- 0
- References
- 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- large language models, iterative optimization, self-improving agents, execution feedback, evidence of learning, execution traces, Atari
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
