CoChat
Sign Up
PreprintCited by 37AI-based Problem Solving and PlanningArtificial Intelligence ApplicationsScientific Computing and Data Management

AFlow: Automating Agentic Workflow Generation

Open access
Zhang, Jiayi, Xiang, Jinyu, Yu, Zhaoyang, Teng, Fengwei, Chen, Xionghui, Chen, Jiaqi, Zhuge, Mingchen, Cheng, Xin +6 more·2024·arXiv
DOI 10.48550/arxiv.2410.10762arXiv 2410.10762
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) have demonstrated remarkable potential in solving complex tasks across diverse domains, typically by employing agentic workflows that follow detailed instructions and operational sequences. However, constructing these workflows requires significant human effort, limiting scalability and generalizability. Recent research has sought to automate the generation and optimization of these workflows, but existing methods still rely on initial manual setup and fall short of achieving fully automated and effective workflow generation. To address this challenge, we reformulate workflow optimization as a search problem over code-represented workflows, where LLM-invoking nodes are connected by edges. We introduce AFlow, an automated framework that efficiently explores this space using Monte Carlo Tree Search, iteratively refining workflows through code modification, tree-structured experience, and execution feedback. Empirical evaluations across six benchmark datasets demonstrate AFlow's efficacy, yielding a 5.7% average improvement over state-of-the-art baselines. Furthermore, AFlow enables smaller models to outperform GPT-4o on specific tasks at 4.55% of its inference cost in dollars. The code is available at https://github.com/FoundationAgents/AFlow.

AI snapshot

Written from the paper’s full text.

TL;DR

AFLOW reformulates agentic workflow generation as a Monte Carlo Tree Search over code-represented workflows, where LLM-invoking nodes are connected by edges and optimized with operators and execution feedback. Across six benchmarks it reports 80.3% average performance, a 5.7% improvement over manually designed methods and 19.5% over automated baselines, while enabling smaller models to outperform GPT-4o on specific tasks at 4.55% of its inference cost.

Key findings

  • On HotpotQA, DROP, HumanEval, MBPP, GSM8K, and MATH, AFLOW with GPT-4o-mini achieves an average of 80.3%, compared with 72.8 for IO, 74.7 for CoT, 76.0 for CoT SC, 75.3 for MedPrompt, 75.1 for MultiPersona, 70.7 for Self Refine, and 67.2 for ADAS; per-dataset scores are 73.5, 80.6, 94.7, 83.4, 93.5, and 56.2 respectively.
  • AFLOW outperforms manually designed methods by an average of 5.7% and surpasses automated workflow optimization by 19.5%; it improves over ADAS on MATH lv5* and MBPP by 57%.
  • Cost analysis reports that executing AFLOW (GPT-4o-mini) with DeepSeek achieves parity with GPT-4o IO at 4.55% of the cost; AFLOW (DeepSeek) with DeepSeek and AFLOW (GPT-4o-mini) with GPT-4o-mini outperform GPT-4o IO at 5.92% and 8.05% of the cost, respectively.
  • On GSM8K, an ablation without predefined operators still reaches 93.1%, surpassing manual designs, and AFLOW autonomously develops ensemble-like structures.

Methods

  • Workflow optimization is formalized as a search over code-represented workflows: nodes are LLM invocations with model, prompt, temperature, and output format; edges are code structures. AFLOW fixes model, temperature, and format, and searches primarily over code edges and prompts using predefined operators such as Generate, Format, Review & Revise, Ensemble, Test, Programmer, and Custom.
  • AFLOW uses a variant of Monte Carlo Tree Search with soft mixed-probability selection, LLM-based expansion, execution evaluation (five runs on the validation set), and experience backpropagation; it stops early if the top-k average score shows no improvement for n consecutive rounds or after N total rounds. The optimizer is Claude-3.5-sonnet; executors include DeepSeek-V2.5, GPT-4o-mini-0718, Claude-3.5-sonnet-0620, and GPT-4o-0513; AFLOW uses 20 iteration rounds, while ADAS uses 30.
  • Experiments use six public benchmarks: full GSM8K, HumanEval, and MBPP; 1,000 samples each for HotpotQA and DROP; and 617 MATH problems from four problem types at difficulty level 5. Data are split into validation and test sets at a 1:4 ratio. Metrics are solve rate for GSM8K/MATH, pass@1 for HumanEval/MBPP, F1 for HotpotQA/DROP, plus token-based cost.

Limitations

  • The paper states it focuses on reasoning tasks with numerical evaluation functions; open-ended tasks without numerical feedback require an LLM-as-judge adaptation and are demonstrated only in case studies on novel generation and academic idea generation, not in the main benchmark evaluation.
  • The paper states it simplifies the search space by fixing key parameters such as model M, temperature τ, and format F, so AFLOW searches primarily over code-represented edges and prompts rather than optimizing those parameters.
  • Evident from the text: evaluation is limited to six benchmarks and specific optimizer/executor model choices, and the operator set is human-designed, though the ablation shows AFLOW can still perform well without predefined operators.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.10762

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 17 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    On MBPP, AFLOW found structures resembling manually designed workflows, including LLM test generation and execution.

    “On the MBPP dataset, AFLOW discovered structures similar to current manually designed workflows, such as test generation and execution by LLMs as seen in Ridnik et al. (2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ridnik et al. (2024) source
  • Interpretationnone given

    AFLOW can reduce human effort by evolving from an empty workflow to a solution closely resembling manually designed workflows such as Ridnik et al. (2024) in code generation.

    “AFLOW demonstrates its ability to reduce human effort by evolving from an empty workflow to a solution highly similar to manually designed workflows like Ridnik et al. (2024) in the code generation scenario.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Ridnik et al. (2024) source
  • Premisecitation

    The diversity and complexity of tasks make it difficult to represent them in a unified yet flexible manner.

    “Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Chen et al., 2021 source

      Section D (p. 25) describes varying levels of requirements, variable dependencies, and the difficulty of defining benchmarks for complex code-synthesis specifications. It does not say that tasks in general are difficult to represent in a unified yet flexible manner; that broader conclusion goes beyond the paper’s discussion. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Cobbe et al., 2021 source

      The Introduction’s Dataset description (page not provided) describes diverse math word problems requiring multiple steps and arithmetic operations. It does not say that differing requirements or dependencies make the tasks difficult to represent in a unified, flexible manner. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Yang et al., 2018 source

      Section 4 (Dataset Analysis; page unavailable) describes HotpotQA questions that require different reasoning operations, including chaining, checking multiple properties, and comparison. Section 5.2 (Results; page unavailable) reports that some question types are harder for the model. The paper does not say that this diversity makes tasks difficult to represent in a unified yet flexible manner; that broader representation claim is unsupported. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation

    Using an LLM as a judge can partly solve the difficulty of evaluating open-ended tasks, since an LLM-as-judge prompt can incorporate human preferences and adaptively score such tasks.

    “By introducing LLM as a judge (Zheng et al., 2023), this challenge can be addressed to some extent, as the LLM-as-judge evaluation prompt can effectively incorporate human preferences to adaptively evaluate open-ended tasks.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al., 2023 source
  • Findingown data

    AFLOW surpasses existing automated workflow optimization approaches by 19.5%.

    “AFLOW outperforms manually designed methods by 5.7% and surpasses existing automated approaches by 19.5%.”
    • Also stated in: § Main Results · plain
      “Workflows optimized by AFLOW outperform all manually designed methods by an average of 5.7% and surpass contemporary automatic workflow optimization work by 19.5%.”
    Quote in the paperSubstantial claim
  • May combine two claims. 41
  • Classified as probably not a substantial claim. 14
  • The quote only partly states this claim. 2

  • May combine two claims.

    Premisecitation

    Existing automated workflow generation methods still depend on initial manual setup and do not achieve fully automated and effective workflow generation.

    “Recent research has sought to automate the generation and optimization of these workflows, but existing methods still rely on initial manual setup and fall short of achieving fully automated and effective workflow generation.”
    • Also stated in: § 1 INTRODUCTION · plain
      “Despite these advancements, full automation has not been achieved.For instance, Khattab et al. (2024) requires manual workflow setup before automated prompt optimization.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Khattab et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yüksekgönül et al., 2024 source

      The Introduction says many compound AI systems were hand-crafted by experts and argues for more principled, automated optimization. Related Work describes efforts to automate prompt and system optimization. Neither passage establishes that existing automated workflow-generation methods require initial manual setup or that they fail to generate effective workflows. (LLM, conf=0.81)

      Experimental — accuracy not yet measured
    • cites Zhuge et al., 2024 source

      The introduction says existing agent approaches require substantial human engineering to define prompting schemes and workflows, supporting the manual-setup concern. The contributions section describes automatic improvements to prompts and agent orchestration, while “Limitation and Future Work” says changing agents’ internal topology remains future work. The paper does not establish the broader claim that existing methods fail to generate effective workflows. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The data are divided into validation and test sets using a 1:4 ratio.

    “Following established practices (Saad-Falcon et al., 2024;Hu et al., 2024) in workflow optimization, we divide the data into validation and test sets using a 1:4 ratio.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Saad-Falcon et al., 2024 source
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Niu et al. (2025) studies ways to adjust workflows dynamically during task execution, improving the system's capacity to handle complex problems by changing workflows in real time according to execution feedback.

    “Meanwhile, Niu et al. (2025) explores methods for dynamically adjusting workflows during task execution, enhancing the system's ability to address complex problems by modifying workflows in real-time based on execution feedback.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Niu et al. (2025) source

  • May combine two claims.

    Premisecitation

    Many real-world tasks have no numerical feedback and are not reasoning tasks.

    “While we emphasized AFLOW's powerful capabilities in solving reasoning tasks with numerical feedback in the main body, there are many tasks in the real world without numerical feedback and non-reasoning requirements (Dai et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dai et al., 2024 source

      The Introduction (p. 1) describes role-playing applications such as emotional companions and social simulations, supporting the point that tasks need not be reasoning tasks. But the paper does not say these tasks lack numerical feedback: Score Quantification (p. 6) distinguishes scoring without a ground truth from its own method of assigning quantitative scores. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Prompt adjustments improved answer quality, consistency, and clarity, especially answer formatting.

    “The following examples demonstrate how adjustments to the prompts enhanced the quality, consistency, and clarity of the answers, with particular emphasis on answer formatting as a key illustration.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The MATH optimal workflow merges programmatic solutions with reasoning strategies, ends with ensemble selection, and formats the answer automatically.

    “It combines programmatic solutions with various reasoning strategies, culminating in an ensemble selection process, and spontaneously formats the answer into the required form.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In a GSM8K ablation, operators improve AFLOW's workflow discovery efficiency and produce incremental gains.

    “An ablation study on GSM8K (Figure 5) shows that operators help AFLOW discover better workflows more efficiently, achieving incremental improvements.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Without operators, AFLOW still achieves 93.1% performance, exceeding manual designs.

    “Notably, even without operators, AFLOW maintains strong performance (93.1%), surpassing manual designs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Each iteration uses a single-step modification: adding an operator or making a targeted prompt change.

    “In each iteration, AFLOW employs a single-step modification, meaning it either adds one operator (rounds 2, 3) or makes a targeted modification to a prompt (rounds 8, 10).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    AFLOW's average score of 80.3 was higher than the average scores of all manually designed and automated workflow optimization methods compared in Table 1.

    “| Ours | 73.5 | 80.6 | 94.7 | 83.4 | 93.5 | 56.2 80.3 |”
    • Also stated in: § ↩→ ↩→ ↩→ ↩→ ↩→ · plain
      “| CoT SC (5-shot) (Wang et al., 2022) | 68.9 | 78.8 | 91.6 | 73.6 | 92.7 | 50.4 76.0 |”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingreasoning

    Round 8 scored lower than baseline Round 1 because it lacked the strict LaTeX formatting constraints from Round 2.

    “This leads to Round 8 scoring lower than the baseline Round 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The eight-iteration optimized workflow generated a 27,000-word novel, showing significant quality and efficiency gains over baselines.

    “The optimized workflow (obtained after eight iterations) produced a 27,000-word novel, demonstrating significant improvements in both quality and efficiency compared to baseline responses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Step 2: Iterative refinement with external knowledge integration max_iterations = 2 for iteration in range(max_iterations): # Retrieve external knowledge knowledge_retrieval_instruction = 'Retrieve relevant information from a knowledge base that can assist in refining the solution.'↩→ knowledge_retrieval_agent = LLMAgentBase(['retrieved_info'], 'Knowledge Retrieval Agent') ↩→ retrieved_results = await knowledge_retrieval_agent(combined_infos, knowledge_retrieval_instruction) ↩→ retrieved_info = retrieved_results[0] # Verify external knowledge verification_instruction = 'Verify the relevancy and accuracy of the retrieved information.'↩→ verification_agent = LLMAgentBase(['verified_info'], 'Verification Agent') verified_results = await verification_agent([taskInfo, retrieved_info], verification_instruction) ↩→ verified_info = verified_results[0] # Refinement phase using verified knowledge refinement_instruction = 'Review and refine the insights provided by other agents using the verified external knowledge.'↩→ refinement_agents = [ LLMAgentBase(['refined_thinking', 'refined_answer'], 'Refinement Agent', role=role, temperature=0.5)↩→ for role in ['Reading Specialist', 'Logic Specialist', 'Generalist'] ] combined_infos_with_verification = combined_infos + [verified_info] async def run_refinement(agent): return await agent(combined_infos_with_verification, refinement_instruction) ↩→ refinement_results = await asyncio.gather(* [run_refinement(agent) for agent in refinement_agents]) ↩→ combined_infos.extend([infofor result in refinement_results for info in result]) # Flattening refinement_results ↩→ # Step 3: Final synthesis agent integrates all refined insights final_decision_instruction = 'Synthesize all refined insights and provide a final answer.' ↩→ final_decision_agent = LLMAgentBase(['thinking', 'answer'], 'Final Decision Agent', temperature=0.3)↩→ final_thinking, final_answer = await final_decision_agent(combined_infos, final_decision_instruction) ↩→ return final_answer}

    ADAS's optimal workflow assigns diverse roles and multiple refinement and summarization steps.

    “As shown in the code in Appendix B.2, the optimal workflow discovered by ADAS assigns diverse roles and multiple steps for refinement and summarization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The MBPP workflow's score is very high and stable.

    “Although this workflow is simple, its score is extremely high and stable”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AFLOW (deepseek) with deepseek and AFLOW (GPT-4o-mini) with GPT-4o-mini exceed GPT-4o IO at 5.92% and 8.05% of the cost, respectively.

    “Executing AFLOW (deepseek) with deepseek and AFlow(GPT-4o-mini) with GPT-4o-mini outperform GPT-4o IO at 5.92% and 8.05% of the cost, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Round 13 had 55 comma-separated answer instances, more than better-performing rounds like Round 5 (43) and Round 9 (42).

    “Statistical analysis shows that Round 13 contains 55 instances of comma-separated answers, notably higher than better-performing rounds such as Round 5 (43 instances) and Round 9 (42 instances).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For MBPP, the best workflow found by AFLOW combines operators with a FIX-CODE prompt and was reached at the fourteenth iteration round.

    “The optimal workflow generated for the MBPP task simply combines operators with an ingenious FIX-CODE PROMPT, achieving the optimal workflow in the iteration at the fourteenth round.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    By automating the design of effective agentic workflows, AFLOW eliminates the human labor costs previously required.

    “By automating the design of effective agentic workflows, AFLOW eliminates the human labor costs previously required.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All workflows are tested thrice on the HumanEval test set, with average results reported.

    “All workflows are tested thrice on the humaneval test set, with average results reported.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    AFLOW executes the blank template five times on the validation dataset and selects a subset of problems with high variance in scores as the final validation set.

    “To optimize computational efficiency, AFLOW then executes the blank template five times on the validation dataset.From these executions, we select a subset of problems that exhibit high variance in scores, which becomes the final validation set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning

    The search space for possible workflows, comprising an immense number of code structures and node configurations, is virtually boundless.

    “Furthermore, the search space for possible workflows, comprising an immense number of code structures and node configurations, is virtually boundless, creating an additional challenge for efficient exploration and optimization.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    ADAS represents workflows with code but fails to generate effective workflows within a limited number of iterations because its linear heuristic search algorithm is inefficient.

    “However, it is constrained by the efficiency of its search algorithm as it relies on overly simplistic representations of experiences in the searching process, making it challenging to discover effective workflows.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The results suggest AFLOW can improve LLM problem-solving while reducing computational cost.

    “These results have highlighted AFLOW's potential for enhancing LLMs' problem-solving capabilities while optimizing computational costs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Neural networks enable adaptive transitions but lack precise control over workflow execution.

    “Neural networks enable adaptive transitions but lack precise control over workflow execution.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The ability to achieve superior performance at lower costs compared to stronger models opens up further possibilities for widespread adoption.

    “Moreover, the ability to achieve superior performance at lower costs compared to stronger models opens up further possibilities for widespread adoption.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Designing and refining agentic workflows demands substantial human effort, which restricts the scalability and adaptability of LLMs to new complex domains and impedes cross-task skill transfer.

    “Designing and refining these workflows requires significant human effort, which limits the scalability and adaptability of LLMs to new, complex domains and hinders their ability to transfer skills across diverse tasks (Tang et al., 2024).”
    • Also stated in: § Abstract · plain
      “However, constructing these workflows requires significant human effort, limiting scalability and generalizability.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tang et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    AFLOW tests each generated workflow 5 times on the validation set, computing mean and standard deviation.

    “We test each generated workflow 5 times on the validation set, computing mean and standard deviation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The operator set O can be easily expanded to enhance search efficiency for various tasks.

    “The operator set O can be easily expanded to enhance search efficiency for various tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Leveraging the tree structure of MCTS to preserve workflow-based exploration experiences enables effective workflow generation and improves search efficiency.

    “Our key idea is to leverage the tree structure of MCTS to preserve workflow-based exploration experiences in N max rounds workflow optimization.When a workflow is revisited, we accurately reuse past successful experiences and avoid failures, enabling effective workflow generation and improving search efficiency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Existing workflow optimization methods that iteratively use past workflow structures to prompt LLMs struggle to guide LLMs towards specific performance metrics due to information loss during accumulation, reducing search efficiency.

    “Existing workflow optimization methods iteratively use past workflow structures to prompt LLMs to discover new structures.However, due to information loss during accumulation (as input tokens increase), this approach struggles to guide LLMs towards specific performance metrics.Combined with the vast search space of code, this reduces search efficiency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Step 2: Iterative refinement with external knowledge integration max_iterations = 2 for iteration in range(max_iterations): # Retrieve external knowledge knowledge_retrieval_instruction = 'Retrieve relevant information from a knowledge base that can assist in refining the solution.'↩→ knowledge_retrieval_agent = LLMAgentBase(['retrieved_info'], 'Knowledge Retrieval Agent') ↩→ retrieved_results = await knowledge_retrieval_agent(combined_infos, knowledge_retrieval_instruction) ↩→ retrieved_info = retrieved_results[0] # Verify external knowledge verification_instruction = 'Verify the relevancy and accuracy of the retrieved information.'↩→ verification_agent = LLMAgentBase(['verified_info'], 'Verification Agent') verified_results = await verification_agent([taskInfo, retrieved_info], verification_instruction) ↩→ verified_info = verified_results[0] # Refinement phase using verified knowledge refinement_instruction = 'Review and refine the insights provided by other agents using the verified external knowledge.'↩→ refinement_agents = [ LLMAgentBase(['refined_thinking', 'refined_answer'], 'Refinement Agent', role=role, temperature=0.5)↩→ for role in ['Reading Specialist', 'Logic Specialist', 'Generalist'] ] combined_infos_with_verification = combined_infos + [verified_info] async def run_refinement(agent): return await agent(combined_infos_with_verification, refinement_instruction) ↩→ refinement_results = await asyncio.gather(* [run_refinement(agent) for agent in refinement_agents]) ↩→ combined_infos.extend([infofor result in refinement_results for info in result]) # Flattening refinement_results ↩→ # Step 3: Final synthesis agent integrates all refined insights final_decision_instruction = 'Synthesize all refined insights and provide a final answer.' ↩→ final_decision_agent = LLMAgentBase(['thinking', 'answer'], 'Final Decision Agent', temperature=0.3)↩→ final_thinking, final_answer = await final_decision_agent(combined_infos, final_decision_instruction) ↩→ return final_answer}

    ADAS's optimal workflow increases problem scale and relies on LLM summarization instead of gradually reducing hops for multi-hop reasoning.

    “Contrary to this, ADAS's optimal workflow actually increases the problem scale, ultimately attempting to use the LLM's summarization ability to synthesize information, rather than gradually reducing the number of hops based on the characteristics of multi-hop reasoning scenarios.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: For multi-hop reasoning tasks, the correct approach is to continuously reduce the problem scale to single-hop reasoning.

  • May combine two claims.

    Interpretationown data

    Cost and performance metrics across iterations indicate AFLOW can produce high-quality long-form content with low resource use.

    “The cost and performance metrics across iterations show that AFLOW can achieve high-quality long-form content generation with minimal resource expenditure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Even without any predefined operators, AFLOW can construct different workflow nodes using the basic Custom operator.

    “Even without any predefined operators, AFLOW can construct different workflow nodes using the basic Custom operator.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    AFLOW randomly partitions the dataset into a validation set (20%) and a test set (80%), with the random seed fixed at 42.

    “Prior to initiating the search process, we randomly partition the dataset into a validation set (20%) and a test set (80%), with the random seed fixed at 42.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This adaptation shows the model's flexibility in customizing workflows for different domains while keeping sophisticated problem-solving structures.

    “This adaptation demonstrates the model's flexibility in tailoring workflows to different problem domains, while maintaining sophisticated problem-solving structures.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Code representation inherently supports all workflow relationships through standard programming constructs, making it the most expressive edge structure.

    “In contrast, code representation inherently supports all these relationships through standard programming constructs.Therefore, we adopt code as our primary edge structure to maximize expressivity.”
    • Also stated in: § Agentic Workflow · plain
      “Code (Hu et al., 2024): A comprehensive representation that can express linear sequences, conditional logic, loops, and incorporate graph or network structures, offering the most precise control over workflow execution for LLMs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    AFLOW's creation of ensemble-like structures without operators shows its capacity for independent workflow design and represents a notable advance toward full automation.

    “Notably, AFLOW autonomously develops ensemble-like structures without operators, demonstrating its capability for independent workflow design and marking a significant step towards full automation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    In AFLOW's MCTS structure, each tree node represents a complete workflow rather than an individual LLM-invoking node, enabling discovery of universal solutions for classes of problems.

    “In our MCTS structure, each tree node represents a complete workflow rather than individual LLM-invoking node, enabling the discovery of universal solutions for classes of problems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The tree-based optimization method enabled exploring several directions at once, developing promising paths while cutting weak branches, which made finding effective operator combinations efficient.

    “The tree structure allowed exploration of multiple optimization directions simultaneously, enabling the development of successful paths while pruning less successful branches, which facilitated the efficient discovery of effective combinations of operators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The tree structure helped reveal patterns distinguishing successful from unsuccessful modifications, exposing shared elements in high-scoring branches, such as Programmer + Custom + ScEnsemble, which guided later optimization.

    “The tree structure made it easier to identify patterns in successful versus unsuccessful modifications, revealing common elements in high-scoring branches, such as the combination of Programmer + Custom + ScEnsemble, which informed future optimization decisions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Feedback on success or failure at each tree node helped show which workflow changes were useful, and the scoring system gave quantitative direction for optimization choices.

    “Each node in the tree represents a specific workflow configuration, and the success/failure feedback at each step helped identify which modifications were beneficial, with the scoring system providing quantitative guidance for optimization decisions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    AFLOW can learn from execution feedback to identify the correct format and automatically format answers.

    “AFLOW can effectively identify the correct format and automatically perform formatting through learning from execution feedback, showcasing the efficacy of this design.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    When a change reduced performance, the tree structure made backtracking easy, permitting exploration of other optimization routes from earlier successful states and preventing entrapment in local optima.

    “When a modification led to decreased performance, the tree structure facilitated easy backtracking, allowing exploration of alternative optimization paths from previous successful states and preventing the process from getting stuck in local optima.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    AFLOW reaches optimal performance in a finite number of iterations when three conditions hold: the evaluation function is bounded, valid workflows are preserved by the code-represented edge structure, and the LLM has a non-zero chance of generating improvements.

    “AFLOW achieves optimal performance within finite iterations under three conditions: • Bounded evaluation function G(W, T ) • Valid workflows maintained by code-represented edge structure • Non-zero probability of LLM generating improvements”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The evaluation function G(W, T) is bounded.
    • The authors state this as an assumption: The code-represented edge structure maintains valid workflows.
    • The authors state this as an assumption: The LLM has a non-zero probability of generating improvements.

  • May combine two claims.

    Interpretationreasoning

    Tree-Structured Experience allows successful modifications to be reused efficiently and repeated failures to be avoided through systematic path tracking.

    “Second, Tree-Structured Experience enables efficient reuse of successful modifications while avoiding repeated failures through systematic path tracking.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Detailed step-by-step solution generation improved solution clarity and comprehensiveness, raising pedagogical value while maintaining accuracy.

    “which enhanced solution clarity and comprehensiveness, ultimately improving the pedagogical value of solutions while maintaining accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    AFLOW can move from any starting workflow to any point in the search space, thereby avoiding local optima.

    “These properties ensure AFLOW can traverse from any initial workflow to any point in the search space, avoiding local optima.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The code-represented edge structure can express every valid relationship between nodes.
    • The authors state this as an assumption: LLM expansion can generate valid workflow modifications with a probability greater than zero.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The multi-agent systems in Zhang et al. (2024a;b); Zhuge et al. (2024) are functionally the same as agentic workflows: both are sequences of LLM-invoking nodes without autonomous capability that run in a fixed logical order.

    “Despite terminological differences, the multi-agent systems described in Zhang et al. (2024a;b); Zhuge et al. (2024) are functionally equivalent to agentic workflows.Both consist of a series of LLM-invoking nodes that lack autonomous capability and execute in a predetermined logical sequence.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al. (2024a;b); Zhuge et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    GSM8K has a larger data volume than other datasets, so the same percentage improvement corresponds to a larger increase in correctly solved samples and avoids fluctuations caused by small data size.

    “Compared to other datasets, GSM8K has a larger data volume, meaning that the same percentage improvement represents a greater increase in correctly solved samples, avoiding fluctuations in improvement due to small data size that could affect comparisons”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The ScEnsemble operator added solution diversity and reliability through ensemble selection, making the system more robust by considering multiple solution approaches.

    “added solution diversity and reliability through ensemble selection, creating a more robust system by considering multiple solution approaches.”
    Quote in the paperSubstantial claim
50 of 50

  • The paper relies on this without support.

    The performance of a workflow on the validation set predicts its performance on the test set.

    “We test each generated workflow 5 times on the validation set, computing mean and standard deviation.”

  • The paper relies on this without support.

    The high-variance problems selected for the validation set are representative of task difficulty.

    “From these executions, we select a subset of problems that exhibit high variance in scores, which becomes the final validation set.”

  • The paper relies on this without support.

    Reasoning tasks have explicit numerical evaluation functions.

    “AFLOW directly executes workflows to get feedback due to explicit evaluation functions in reasoning tasks.”

  • The paper relies on this without support.

    For multi-hop reasoning tasks, the correct approach is to continuously reduce the problem scale to single-hop reasoning.

    “However, for multi-hop reasoning tasks, the correct approach is to continuously reduce the problem scale to single-hop reasoning.”

  • The authors state this as an assumption.

    The code-represented edge structure can express every valid relationship between nodes.

    “• Code-represented edge structure can express all valid node relationships”

  • The authors state this as an assumption.

    LLM expansion can generate valid workflow modifications with a probability greater than zero.

    “• LLM expansion generates valid workflow modifications with non-zero probability”

  • The authors state this as an assumption.

    The evaluation function G(W, T) is bounded.

    “• Bounded evaluation function G(W, T )”

  • The authors state this as an assumption.

    The code-represented edge structure maintains valid workflows.

    “• Valid workflows maintained by code-represented edge structure”

  • The authors state this as an assumption.

    The LLM has a non-zero probability of generating improvements.

    “• Non-zero probability of LLM generating improvements”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent works demonstrate that coding agents can be improved through meta-learning of their designs, such as prompts, workflows, and tools, which are implemented in their code repository.

    “Recent works [53, 149] demonstrate that such agents can be improved through meta-learning of their designs (e.g., prompts, workflows, and tools), which are implemented in their code repository.”

    From Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents (2025) · cites this paper as [149]

    The Automated Agentic Optimization section (page unavailable) describes automatic prompt and workflow optimization, and Main Results (page unavailable) reports improved performance on benchmarks that include code tasks. The paper does not show optimization of agents’ tools or designs implemented in a code repository; its use of code is to represent and modify workflows. Thus it supports the workflow-improvement part, but not the full set of examples attributed to it. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • MAS-GPT outperforms math-specific AFlow on the MATH dataset by 3.53%.

    “As a general method, our MAS-GPT even outperforms math-specific AFlow on the MATH dataset by 3 . 53% !”

    From MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems (2025) · cites this paper as Zhang et al., 2024

    Table 1 reports AFlow’s 56.2% solve rate on its MATH test set, and the Conclusion describes a workflow tailored to the MATH task. The paper does not evaluate MAS-GPT or establish that it outperforms AFlow by 3.53%; that comparison is not supported by this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • AFlow needs to call APIs of powerful proprietary LLMs, such as Claude-3.5Sonnet, 10 times per query and depends on a hold-out validation set.

    “while AFlow needs to call the APIs of powerful proprietary LLMs, such as Claude-3.5Sonnet (Anthropic, 2024), 10 times per query and depends on a hold-out validation set.”

    From MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems (2025) · cites this paper as Zhang et al., 2024

    The Datasets and Implementation Details sections (pages not provided) say AFlow uses Claude-3.5-sonnet as an optimizer, accesses its models through APIs, and splits data into validation and test sets. The Expansion section says each generated workflow is tested five times on the validation set; it does not establish a requirement of ten proprietary-LLM API calls per query. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • The paper's unified representation of MAS is motivated by the observation that all existing LLM-based MAS methods are ultimately implemented as code.

    “This unified representation is motivated by the observation that all existing LLM-based MAS methods are ultimately implemented as code, encompassing the definition of agents' prompts, LLM calls, and inter-agent interactions (Qian et al., 2024a; Hu et al., 2024; Zhang et al., 2024).”

    From MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems (2025) · cites this paper as Zhang et al., 2024

    In the Agentic Workflow section (page unavailable), AFlow adopts code to represent workflow relationships because it can express sequences, conditions, loops, and graph structures. Its Additional information section (page unavailable) treats some multi-agent systems as functionally equivalent to agentic workflows. Neither passage says that all existing LLM-based MAS methods are ultimately implemented as code, or gives that universal observation as the reason for the representation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • AFlow casts coderepresented workflow optimization as Monte Carlo tree search.

    “AFlow casts coderepresented workflow optimization as Monte Carlo tree search [8].”

    From HELIX: Model-Harness Co-evolution for Recursive Self-Improvement (2026) · cites this paper as [8]

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into state-of- the-art pipelines. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.

Cited by

Sorted by citations

Related

Authors
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., Zheng, B., Liu, B., Luo, Y., & Wu, C.
Affiliations
  • The Hong Kong University of Science and Technology (Guangzhou) (CN)
  • Renmin University of China (CN)
  • Nanjing University (CN)
  • Fudan University (CN)
  • King Abdullah University of Science and Technology (SA)
  • Université de Montréal (CA)
  • Hong Kong University of Science and Technology (HK)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.10762
arXiv
2410.10762
OpenAlex
W4403571374
Semantic Scholar
53132ea6c107479d4557631299d3ed525109b464
Cited by
7
References
51
Citations by year
2026: 1 · 2025: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
agentic workflows, workflow generation, workflow optimization, large language models, Monte Carlo tree search, code search, LLM agents, execution feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review