The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Developing Multi-Agent LLM Applications Through Continuous Human-LLM Co-Programming
Abstract
The rapid advancement of Large Language Models (LLMs) has opened new possibilities for intelligent multi-agent systems capable of autonomously performing complex tasks. To build such systems, LLMs can be leveraged for task-solving, tool interaction, and code generation but at the same time their costs and unpredictability have to be properly managed. To do so this paper introduces COPMA, a model-based approach to enabling continuous human-LLM co-programming of multi-agent LLM applications. COPMA uses feature-block models to track application features and their implementations as agents and code blocks. Supported by co-programming patterns, de-velopers are guided in constructing, refining, and refactoring feature implementations via trial-and-errors with LLM agents, leveraging their feedback, suggestions, and code examples. The patterns guide the shift of feature implementations between agents and code to balance flexibility, predictability, and cost. Our experience in developing LLM agents for collecting and reviewing medical research papers demonstrates that human-LLM co-programming can reduce development effort to enable rapid prototyping of multi-agent LLM applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
Large Language Model based Multi-Agents: A Survey of Progress and Challenges
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X. (2024) · arXiv · doi:10.48550/arxiv.2402.01680 · Cited by 1,713
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404 · cited 1×
“However, the construction of functioning and robust multi-agent applications still requires heavy programming effort, and effective methods and practices for this new programming paradigm do lack [13], [14].”
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · cited 1×
“However, the inherent unpredictability of LLMs also introduces the risk of unexpected behaviors [6]–[9].”
Are Emergent Abilities of Large Language Models a Mirage?
Schaeffer, R., Miranda, B., & Koyejo, S. (2023) · arXiv · doi:10.48550/arxiv.2304.15004 · Cited by 200
(2023) · doi:10.52202/075280-2425 · Cited by 200
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Liu, Y., Lo, S. K., Lu, Q., Zhu, L., Zhao, D., Xu, X., Harrer, S., & Whittle, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112278 · Cited by 51
Flagged first, then by citations
Research works printed but not linked: 34
- Liu, 2024, Reason for future, act for now: A principled architecture for autonomous lIm agents, ICML’24
- Talebirad, 2023, Multi-agent collaboration: Harnessing the power of intelligent lIm agents, arXiv preprint
- Liu, 2023, Agentbench: Evaluating IIms as agents, arXiv preprint
- 10.24963/ijcai.2024/890 · DOI · Paper page
- Wu, 2023, Autogen: Enabling next-gen lIm applications via multi-agent conversation framework, arXiv preprint
- Xie, 2023, Openagents: An open platform for language agents in the wild, arXiv preprint
- 2024, Agentgpt: a tool to configure and deploy autonomous ai agents
- Dibia, 2024, Multi-agent lIm applications - a review of current research, tools, and challenges, designingwithml.com
- 10.1145/3695988 · DOI · Paper page
- Achiam, 2023, Gpt-4 technical report, arXiv preprint
- 10.1109/MS.2020.2993662 · DOI
- 10.1145/3638247 · DOI
- 10.1145/3487043 · DOI
- 10.1007/s10515-024-00426-z · DOI
- 10.1109/ICSE-FoSE59343.2023.00008 · DOI · Paper page
- 10.1007/b96018 · DOI
- 10.1017/S0269888913000088 · DOI
- 10.1007/11787044_22 · DOI
- 10.1007/978-3-540-75524-1_12 · DOI
- 10.1007/978-3-540-74949-3_2 · DOI
- 10.1016/j.cl.2017.06.001 · DOI
- 10.1016/j.simpat.2017.11.006 · DOI
- Kardas, 2006, Metamodeling of semantic web enabled multi agent systems, Multiagent Systems and Software Architecture, 2006, 79
- 10.1142/S0218843009002014 · DOI
- 10.1016/j.jhep.2017.07.022 · DOI · Paper page
- 10.1038/d41586-024-03676-9 · DOI
- Canese, 2013, Pubmed: the bibliographic database, The NCBI handbook, 2
- 10.1162/qss_a_00022 · DOI
- 2024, Neo4j: Graph database and analytics
- 10.1016/j.is.2010.01.001 · DOI
- 10.1109/MS.2002.1020288 · DOI
- 2024, Cyper query language
- 10.1126/scitranslmed.abb3107 · DOI
- 10.55662/JST.2023.4605 · DOI
Cited by
Sorted by citations
Nedisan Videsjorden, A., Song, H., Goknil, A., Roman, D., & Soylu, A. (2026) · ACM Transactions on Internet of Things · doi:10.1145/3772077 · Cited by 5
“The evolution of LLMs from single-agent models to multi-agent frameworks has enabled collaborative problem-solving [26, 83].”
Self-citation
Metadata
- Authors
- Song, H., Goknil, A., Jiang, X., Melum, E., Joe, H., Gazzotti, C., Frascolla, V., Videsjorden, A. N., & Nguyen, P.
- Affiliations
- SINTEF Digital (NO)
- SINTEF Digital,Oslo,Norway
- Oslo University Hospital (NO)
- Seoul National University (KR)
- University of Modena and Reggio Emilia (IT)
- Intel (Germany) (DE)
- ORCID
- Venue
- 2025 IEEE/ACM 4th International Conference on AI Engineering – Software Engineering for AI (CAIN)
- Publisher
- IEEE
- Published
- 2025-04-27
- Pages
- 42-47
- OpenAlex
- W4411337603
- Cited by
- 3
- References
- 9
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 3.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) (publishedVersion)
- Funders
- International Research Development Program of the National Research Foundation of Korea (RS-2023–00268071)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- large language models, rapid prototyping
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
