The DOI matches its DataCite record.
Specifications: The missing link to making the development of LLM systems an engineering discipline
Open accessAbstract
Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Self-verification may miss flaws, rely on vague conditions, or be impractical.
“However, it may miss flaws, rely on vague conditions, or be impractical [36].”
From Why Do Multi-Agent LLM Systems Fail? (2025) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 12
- F. P. B. Jr., The Mythical Man-Month: Essays on Software Engineering. Reading, Massachusetts: Addison-Wesley, 1st ed., 1975.
- Jerome Goddard, "Hallucinations in ChatGPT: A cautionary tale for biomedical researchers," in The American Journal of Medicine, Elsevier Inc, 2023. https://www.amjmed.com/action/showPdf?pii=S0002-9343% 2823%2900401-1.
- X. Zhu, S. Wen, S. Camtepe, and Y. Xiang, "Fuzzing: A survey for roadmap," ACM Comput. Surv., vol. 54, Sept. 2022.
- W. B. Demilie and F. G. Deriba, "Detection and prevention of sqli attacks and developing compressive framework using machine learning and hybrid techniques," Journal of Big Data, vol. 9, no. 1, p. 124, 2022.
- E. Çınlar, Probability and Stochastics. Graduate Texts in Mathematics, Springer New York, 2011.
- L. Kleinrock, Theory, Volume 1, Queueing Systems. USA: Wiley-Interscience, 1975.
- Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d'Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals, "Competition-level code generation with alphacode," Science, vol. 378, p. 1092-1097, Dec. 2022.
- Y. LeCunn, "A path towards autonomous machine intelligence," 2022. https://openreview.net/pdf?id= BZ5a1r-kVsf.
- V. Pallagani, K. Roy, B. Muppasani, F. Fabiano, A. Loreggia, K. Murugesan, B. Srivastava1, F. Rossi, L. Horesh, and A. Sheth1, "On the prospects of incorporating large language models (LLMs) in automated planning and scheduling (APS)," 2024. https://arxiv.org/html/2401.02500v2.
- R. M. Hierons, K. Bogdanov, J. P. Bowen, R. Cleaveland, J. Derrick, J. Dick, M. Gheorghe, M. Harman, K. Kapoor, P. Krause, G. Lüttgen, A. J. H. Simons, S. Vilkomir, M. R. Woodward, and H. Zedan, "Using formal specifications to support testing," 2009. https://dl.acm.org/doi/10.1145/1459352.1459354. · DOI
- G. C. Necula, "Proof-carrying code," in POPL '97: Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, 1997.
- S. G. Patil, T. Zhang, V. Fang, R. Huang, A. Hao, M. Casado, J. E. Gonzalez, R. A. Popa, I. Stoica, et al., "Goex: Perspectives and designs towards a runtime for autonomous llm applications," arXiv preprint arXiv:2404.06921, 2024. · arXiv
Cited by
Sorted by citations
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
“Do not distribute.”
Self-citation
Metadata
- Authors
- Stoica, I., Zaharia, M., Gonzalez, J., Goldberg, K., Sen, K., Zhang, H., Angelopoulos, A., Patil, S. G., Chen, L., Chiang, W.-L., & Davis, J. Q.
- Affiliations
- University of California, Berkeley (US)
- University of California San Diego (US)
- Microsoft (United States) (US)
- Stanford University (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2412.05299
- OpenAlex
- W4405898991
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- specifications, large language models, modular systems, system reliability, LLM agents, structured output, process supervision, test-time compute, component-based system design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
