Self-collaboration code generation via ChatGPT

“Recent studies have shown that well-designed prompting methods, such as chain-of-thought prompting (CoT) [Wei et al., 2022] and zero-shot CoT [Kojima et al.”

“Add 𝑠 𝑡 to B .”

“In this section, we further analyze the errors generated by self-collaboration and the cost of self-collaboration compared to other prompting approaches.”

“This kind of baseline includes: —Chain-of-thought (CoT) [57] generates a chain of thought for each question and then generates the corresponding code.”

“The benefits of collaborative teamwork are manifold: (1) It breaks down complex tasks into smaller subtasks, making the entire code generation process more efficient and controllable.”

“Consequently, it is prevalent to control LLM generation using instructions or prompts [11, 40, 41].”

“Recently, code generation has made substantial advancements in both academic and industrial domains [8, 16, 30, 49].”

“However, considering the high labor expenses of software development teams, the cost-effectiveness of self-collaboration is obvious.”

“In particular, large language models (LLMs) have achieved excellent performance and demonstrate promising potential on code generation tasks [15, 18, 38, 64].”

“Meanwhile, since hand-crafted instructions or prompts may not always be optimal, some researchers have explored automated instruction or prompt generation approaches [24, 44, 51, 65].”

“The benefits of collaborative teamwork are manifold: (1) It breaks down complex tasks into smaller subtasks, making the entire code generation process more efficient and controllable.”

“The pseudocode of our self-collaboration framework is outlined in Algorithm 1.”

“The task is defined as T : 𝑥 → 𝑦 .”

“The software development lifecycle (SDLC) breaks up the process of creating a software system into discrete stages, including requirement analysis, planning and design, coding, testing, deployment, and maintenance.”

“, 2023] and HuggingGPT [Shen et al., 2023] explore the collaboration of LLMs with other models, specifically the LLMs are used as decision centers to control and invoke other models to handle more domains, such as vision, speech, and signals.”

“(2) This setting is mainly suited for function-level code generation and proves challenging to extend to file-level or repository-level code generation.”

“The task is defined as T : 𝑥 → 𝑦 .”

“MBPP (sanitized version) [Austin et al., 2021] contains 427 manually verified Python programming tasks, covering programming fundamentals, standard library functionality, and more.”

“Meanwhile, since hand-crafted instructions or prompts may not always be optimal, some researchers have explored automated instruction or prompt generation approaches [24, 44, 51, 65].”

“In particular, large language models (LLMs) have achieved excellent performance and demonstrate promising potential on code generation tasks [15, 18, 38, 64].”

“CAMEL [28] explores the possibility of interaction between two LLMs.”

“We introduce the classic waterfall model [Petersen et al., 2009] from SDM into self-collaboration framework to make the teamwork for code generation more efficient.”

“In this section, we outline the most relevant directions and associated papers of this work to highlight our research’s innovative and advanced nature.”

“Recently, VisualGPT [58] and HuggingGPT [50] explore the collaboration of LLMs with other models, specifically the LLMs are used as decision centers to control and invoke other models to handle more domains, such as vision, speech, and signals.”

“Self-planning [Jiang et al., 2023] introduces a planning stage prior to code generation.”

“CoderReviewer [Zhang et al., 2022] introduces an additional Reviewer model to enhance the coder model, which generates requirements based on the code and calculates the maximum likelihood to rank the generated code.”

“Therefore, self-collaboration may require the following abilities for LLMs, including strong domain-specific expertise ability for role-playing, strong in-context learning ability to follow instructions, and strong reasoning ability to solve problems effectively.”

“Additionally, the work [Xia and Zhang, 2023] successfully implemented conversational APR using ChatGPT.”

“Following the settings in CodeT [7], we use NL in APPS as a comment and then concatenate the signature “ def solution(stdin : str) → str : ” as input to the model.”

“Self-debugging [Chen et al., 2023] teaches LLM to perform rubber duck debugging after code generation.”

“MBPP-ET and HumanEval-ET [Dong et al., 2023] are expanded versions of MBPP and HumanEval with over 100 additional test cases per task.”

“Teamwork through division of labor (DOL) , interaction, and collaboration to solve complex problems, has been theorized to play an important role in dealing with complexity, as posited in both teamwork theory [5, 26] and software engineering practice [4, 13, 34].”

“In a comprehensive study conducted by the work [Xia et al., 2022], the direct application of LLMs for APR was explored and it was shown that LLMs outperform all existing APR techniques by a substantial margin.”

“In particular, large language models (LLMs) have achieved excellent performance and demonstrate promising potential on code generation tasks [15, 18, 38, 64].”

“A straightforward way to implement collaborative teamwork entails training different models to handle the corresponding subtasks, subsequently conducting joint training to foster mutual understanding of behaviors to assemble them into a team [Schick et al., 2022].”

“This is a critical task since the behavior of LLMs depends heavily on the inputs they receive [6, 17, 31].”

“The work [Zelikman et al., 2022] applied LLMs to code generation and demonstrated significant improvement in the pass rate of generated complex programs.”

“Multi-agent collaboration refers to the coordination and collaboration among multiple artificial intelligence (AI) systems or the symbiotic collaboration between AI and human, working together to achieve a shared objective [Smoliar, 1991].”

“ChemCrow [75], Boiko et al. [115], Grossmann et al. [122] Natural Science Education ChemCrow [75], CodeHelp [120], Boiko et al. [115], MathA-gent [117], Drori et al. [118] Engineering”

“LLMs have been used for various coding-related tasks including code generation and code recommendation [20], [21], [22], [23].”

“Dong et al. [72] treated LLMs as agents, letting multiple LLMs play distinct roles in addressing code generation tasks collaboratively and interactively.”

“Through ongoing interactions, agents continuously assess and enhance their knowledge base, [32], ChatEDA [303], LIBRO [304], PENTESTGPT [305] Industrial Engineering Mehta et al. [2], Xia et al. [3], Li et al. [4] Embodied Agents SayCan [8], Inner Monologue [9], TidyBot [10], RoCo [34], CoELA [35]…”

“Follow-up works explore how to utilize zero-shot discrete prompt for domain adaptation in sentiment analysis [60], image classification [38], semantic segmentation [35], and rumor detection [82].”

“Self-collaboration (Dong et al., 2023b) proposes a framework where different LLMs act as analyst, coder, and tester to cooperatively generate code for complex tasks, achieving better performance than directly using a single LLM. AlphaCodium (Ridnik et al., 2024) iteratively refines code based on…”

“In these works, such LLM-agent team is often set up statically with predefined roles for each agent, such as introducing programmer and tester to communicate sequentially in a fixed order to improve code generation (Dong et al., 2023).”

“This feedback initiates an iterative cycle, where the Programmer refines the code or the Debugger resolves identified issues, ensuring that the final code meets the desired standards and performs as expected [21, 31, 69, 72, 81, 88, 96, 102, 132].”

“This feedback initiates an iterative cycle, where the Programmer refines the code or the Debugger resolves identified issues, ensuring that the final code meets the desired standards and performs as expected [21, 31, 69, 72, 81, 88, 96, 102, 132].”

“”) to inspire ChatGPT’s capability of test generation, which is a common prompt optimization strategy [53], [64]; and (ii) a task-description instruction (i.”

“…the quality of code generated by LLMs (Zhang et al. 2023a; Wang et al. 2023d; Barke et al. 2023a; Mouselinos et al. 2023; Lahiri et al. 2022; Dong et al. 2023; Jiang et al. 2023a), addressing security and reliability concerns (Poesia et al. 2022; Zhu and Zhang 2023; Ke et al. 2023),…”

“Many previous literature mentioned the use of role-playing to enhance or surpass the performance of ChatGPT [30, 33, 61, 68, 105].”

“(A1, A2, A5, and A6) • Collaborative/adversarial agent interactions : Like many collaborative agent systems (Dong et al., 2023), agents in AutoGen can share information and knowledge, to complement each other’s abilities and collectively arrive at better solutions.”

“LLMs pre-trained on code have demonstrated outstanding results in code generation tasks [5, 7, 8, 14, 29], which are fundamentally dependent on exceptional code comprehension.”

“In Python, the top three issues are ConsiderUsingEnumerate (213 out of 2033 programs, 10.5%), NoElseReturn (161 out of 2033 programs, 7.9%), and UnusedVariable (103 out of 2033 programs, 5.1%).”

“Dong et al. [14] proposed a self-collaboration framework for LLMs to enhance their capability of solving coding problems.”

“Many studies have utilized HumanEval to test the performance of code generation models [101].”

“5 with RLHF) [3] has shown superior performance in solving various complex tasks (e.g., program synthesizes [61], program repair [62]) by only following the user’s instructions and feedback through a dialogue.”

“…scenarios, some works focus on improving the prompt technologies by introducing specific patterns, e.g., Structured Chain-of-Thought (Li et al., 2023a), Self-planning (Jiang et al., 2023), Self-debug (Chen et al., 2023), Self-collaboration (Dong et al., 2023), and AceCoder (Li et al., 2023c).”

“Dong et al. (2023b) assigned three roles: analyst, coder, and tester to three distinct “GPT-3.”

“Moreover, recent works (Wang et al., 2023e; Fu et al., 2023; Liang et al., 2023; Schick et al., 2022; Dong et al., 2023; Park et al., 2023; Liu et al., 2023) have utilized a multi-role dialogue to collaborate or debate with each other for a more comprehensive response.”

“Dong et al. [53] introduce the concept of software development life cycle and propose a self-collaboration framework that leverages different ChatGPT conversations to play different roles (e.g., analyst, developer, and tester), collaborating to generate code.”

“Dong et al. [176] presents a self-collaboration framework where LLM agents function as distinct “experts,” each managing specific subtasks to facilitate autonomous collaborative code generation.”

“…Large Language Models (LLMs), pre-trained on billions of text/code tokens, bypass the need for re-training/fine-tuning but demonstrate the powerful generality of various code-related tasks, such as code generation [7, 11, 23, 25, 28, 52], program repair [12, 48], and code summarization [2, 14].”

“Dong et al. [21] proposes a self-collaboration method to advance LLMs in complex coding tasks by employing multiple LLMs as distinct experts and making them interact with each other.”

“…scenarios, some works focus on improving the prompt technologies by introducing specific patterns, e.g., Structured Chain-of-Thought (Li et al., 2023a), Self-planning (Jiang et al., 2023), Self-debug (Chen et al., 2023), Self-collaboration (Dong et al., 2023), and AceCoder (Li et al., 2023c).”

“Dong et al. [36] point out that code generation research should not be confined to programming alone, and suggest adopting multiple LLM agents to simulate different development team roles, such as analysts who break down user requirements, developers who generate or update code, and testers who…”

“Additionally, some social complex tasks, such as court simulations and software development, become possible through the collaboration between LLM-based agents [8, 23, 27].”

“…in several domains, such as conversational AI (Alessa and Al-Khalifa, 2023; Köpf et al., 2023; Chen et al., 2023), education (Küche-mann et al., 2023; Eshghie and Eshghie, 2023), code programming (Dong et al., 2023; Yeti¸stiren et al., 2023) and healthcare (Zhao et al., 2023; Yang et al., 2023).”

“Furthermore, research has shown that LLMs have remarkable performance on various code intelligence tasks [35], including code generation [25, 27, 31, 37, 43, 74], code completion [16, 23, 24, 44, 45, 54], and code summarization [9, 14, 30, 34, 47, 48].”

“In addition to the above environments, several research also proposes general frameworks applicable to different environments (§2.2.6).”

“Prompt Learning is one of the most important techniques for achieving this goal [11, 28, 31, 40, 47].”

“(3) Software implementation is the detailed creation procedure of the software to implement the design [16; 1; 6; 29; 11].”

“Learning from software repositories has opened up a myriad of LLM applications in software development tasks, such as code search [37], code quality [5, 38, 39], repair [19, 33, 40, 44], program comprehension [24], generation [14, 17], completion [28], and translation [21].”

“Additionally, recent research explores using LLMs for evaluating literary translations (Yan et al., 2024; Zhang et al., 2024).”

“Language Models (LMs) have demonstrated remarkable effectiveness in generating content that closely resembles human performances across diverse tasks (Tan et al. 2023; Dong et al. 2023; Liu et al. 2023).”

“5 with RLHF) [3] has shown superior performance in solving various complex tasks (e.g., program synthesizes [61], program repair [62]) by only following the user’s instructions and feedback through a dialogue.”

“Digital environments .”

“It should be noted that in the generation process, Designer agent will be called only once to generate diverse test inputs, but Calculator agent will be called for each test input.”

“[ Dong et al. , 2023b] makes LLMs work as distinct “experts” for sub-tasks in software development, autonomously collaborating to generate code.”

“Tremendous progress has been made by treating programming”

“5 62.2 52.4 – Self-Debugging (Chen et al., 2023b) GPT-4 – 80.6 – Self-Collaboration (Dong et al., 2024) GPT-3.”

“MetaGPT [1], ChatDev [2], and self-collaboration [3] pre-set various roles and corresponding responsibilities to foster collaboration among agents.”

“This model demonstrates a significant advancement in solving multiple tasks, including code generation (Dong et al., 2023; Sakib et al., 2023; Feng et al., 2023), machine translation (Jiao et al., 2023; Peng et al., 2023), even game playing (Wu et al., 2024).”

“In line with previous works [7, 10, 22, 30], we use the greedy pass@1 [6, 49] to assess the functional correctness of the generated program.”

“Hong et al. (2024); Dong et al. (2023) adopt a shared information pool to reduce overhead.”

“These agents are widely applied in various scenarios and applications, such as autonomous software development [50–52], serving as tools to assist humans in accomplishing specific tasks [53–55], and acting as planners to break down complex tasks and generate executable plans [56–58].”

“In N=50 trials, GPT-4 produces functional I2C interfaces 66% of the time.”

“LLM-based agents employ various planning strategies, including single [23] or multiple planners [24], single [25] or multi-turn planning [26], and single [27] or multi-path planning [28].”

“[14].”

“LLMs have become increasingly prevalent in various social-influencing domains such as education (Kasneci et al., 2023; AlAfnan et al., 2023) and the technology industry (Dong et al., 2023), playing a more and more important role in social influence.”

“Additionally, it has revealed some emergent behaviors exhibited by agents in multi-agent collaborations, such as volunteer and conformity behaviors.”

“…the advent of ChatGPT 1 , an advanced NLP model derived from GPT-3, has showcased remarkable capabilities in generation (Maddigan and Susnjak, 2023; Dong et al., 2023; Liu et al., 2023a), contextual understanding (Bang et al., 2023; Gao et al., 2023b; Amin et al., 2023), and reasoning (Liu et al.,…”

“Follow-up works explore how to utilize zero-shot discrete prompt for domain adaptation in sentiment analysis [60], image classification [38], semantic segmentation [35], and rumor detection [82].”

“Large Language Model agents have shown remarkable performance in various tasks through mutual collaboration (Chen et al., 2023; Hua et al., 2023; Cohen et al., 2023; Zhou et al., 2023; Li et al., 2023b,a; Chan et al., 2023; Dong et al., 2024; Qian et al., 2023).”

“Nevertheless, generating high-quality and reliable code remains a formidable task [12]–[15].”

“The profiling module defines an agent’s operational persona (e.g., developer, advisor, or task-specific role), conditioning its behavior and role policy through static profiles defined by experts or dynamic generative mechanisms [21, 138].”

“There has been a growing interest in using Generative AI techniques for software engineering tasks, including requirements engineering, design, and testing [1, 3, 19-22].”

“GPT-4 currently provides state-of-the-art performance in NLP tasks such as natural language translation (Li et al., 2023b) and even translation to structured representations (Olmo et al., 2021).”

“Based on this process, some end-to-end software development agents [4], [187], [191], [194] further extend the traditional waterfall process by including iterations in specific phases to ensure the high quality of the generated content.”

“To evaluate LLMs on code generation tasks, the most common metric is Pass@k [28], where k = 1 is used to measure the rate of passed functions that the LLM generated on the first attempt [24] (e.g., by running a test suite).”

“Since the automatic performance measures can only reflect the lexical similarity be-tween the generated smart contract comment and the ground-truth smart contract comment, we finally conduct a human study to evaluate the quality of the generated comments.”

“5-turbo [32], ReAct [51], Reflexion [42], ToT [50], RAP [14], Self-Edit [52], Self-Planning [19], Self-Debugging [4], Self-Collaboration [10], SCoT [27], CodeChain [22], INTERVENOR [44], CodeCoT [16], PaLM Coder [8], Claude-instant-1 [29], GPT-4 [32], MapCoder [18], CodeTree [26], and DebateCoder…”

“Dong et al. [1] and Qian et al. [3] suggested a cooperative approach where LLMs like ChatGPT adopt multiple roles (analyst, coder, and tester) to collaboratively tackle code generation tasks.”

“With the rapid development of Large Language Models (LLMs) like the GPT series [28], [29], [30], [31] and specialized models such as Fuyu-8B [32], LLM-based agents have demonstrated powerful capabilities across numerous domains [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43].”

“The emergence of LLM agents has sparked interest in multi-agent collaboration systems [9, 10, 29], where multiple agents combine capabilities to tackle complex tasks [1, 11].”

“They can be used for a variety of tasks, including natural language processing (NLP), text generation, and even code generation [7].”

“This demographic benefits from a personal coding assistant to directly stimulate new ideas for addressing problems [9,26-27, 29 ] in”

“Using AI itself to provide feedback is also an option; notably, AI-based feedback is also used outside the RL paradigm, e.g., to provide verbal feedback to be appended to prompts (Shinn et al., 2023) or to collaborate with other LLMs at inference time (Dong et al., 2023; Du et al., 2023).”

“In 2022, OpenAI launched ChatGPT, a generative AI chat-bot based on LLMs, which is suitable for natural language processing tasks, such as question answering [19], [20], conversational agents [21], [22], [23], and code completion [24], [25].”

“LLMs have shown promising abilities in code-related tasks, including code generation (Chen et al. 2021a; Dong et al. 2024; Li et al. 2023a,b; Yang et al. 2024a), program repair (Fan et al. 2023; Xia et al. 2023), and code summarization (Ahmed and Devanbu 2023).”

“…Code summarization: [45]–[47]The model generates a short natural language summary from a source code snippet (SC-to-NL), Code generation: [48]–[50]The model generates the corresponding code snippet from a natural language description (NL-to-SC), and Code translation: [51]–[53] The model…”

“Thus, pre-trained methods with big data-driven usually employ Seq2Seq approaches [2, 8, 11, 14, 23, 38].”

“Dong et al. [61] set up an experimental design aimed at mimicking human teamwork via ChatGPT: The authors assemble a virtual team composed of three ChatGPT instances instructed to cover different roles, namely analyst, programmer, and tester.”

“Research in Large Language Models (LLMs) has made significant advancements in fields such as Natural Language Processing [11], Computer Vision [29], Code Generation [15], and various AI tasks [28, 31].”

“Software engineering is no exception to the adoption of LLMs where LLMs such as ChatGPT have been proven to be essential for understanding and generating code [30], documentation [32] and interactions in natural language [20].”

“Efforts are increasingly focused on leveraging LLMs for complex tasks like planning and reasoning (Wang et al., 2023c; Wu et al., 2023; Dong et al., 2023).”

“LLMs have shown great potential in software engineering tasks like code generation [56], [57], [58], program repair [59], [60], and code summarization [61], [62].”

“In recent years, the LLM-based code generation task has drawn widespread attention [24], [25], [36]–[38], [52]–[54], [75], [76].”

“For instance, MetaGPT [30], ChatDev [59] and [25] explore software development by dividing roles among agent groups within a wa-terfall model.”