Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference
Open accessAbstract
Large Language Models (LLMs) have sparked significant interest in their generative capabilities, leading to the development of various commercial applications. The high cost of using the models drives application builders to maximize the value of generation under a limited inference budget. This paper presents a study of optimizing inference hyperparameters such as the number of responses, temperature and max tokens, which significantly affects the utility/cost of text generation. We design a framework named EcoOptiGen which leverages economical hyperparameter optimization and cost-based pruning. Experiments with the GPT-3.5/GPT-4 models on a variety of tasks verify its effectiveness. EcoOptiGen is implemented in the `autogen' package of the FLAML library: \url{https://aka.ms/autogen}.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Preset model characteristics, such as temperature, token length, fine-tuning availability, penalties, add-on availability, language, and layers, further impact LLM functionality.
“preset model characteristics, such as temperature, token length, fine-tuning availability, penalties, add-on availability, language, and layers, further impact their functionality.157,158,159”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [157]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
EcoAssistant: Using LLM Assistant More Affordably and Accurately
Zhang, J., Krishna, R., Awadallah, A. H., & Wang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.03046 · Cited by 8 · see citing works
Self-citationAutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Park, S., Kim, W., Tan, R., Zhang, J., Han, W.-S., Gao, P., Park, C., Yao, Y., Fu, R., Nallipogu, E., et al. (2026) · arXiv · doi:10.48550/arxiv.2608.23041
- Authors
- Wang, C., Liu, S. X., & Awadallah, A. H.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2303.04673
- OpenAlex
- W4323717344
- Cited by
- 13
- References
- 0
- Citations by year
- 2025: 9 · 2024: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- hyperparameter optimization, large language models, inference cost, text generation, temperature, FLAML, GPT-3.5, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
