CoChat
Sign Up
PreprintCited by 2Speech and dialogue systemsTopic ModelingArtificial Intelligence Applications

APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

Open access
Prabhakar, Akshara, Liu, Zuxin, Zhu, Ming, Zhang, Jianguo, Awalgaonkar, Tulika, Wang, Shiyu, Liu, Zhiwei, Chen, Haolin +7 more·2025·arXiv
DOI 10.48550/arxiv.2504.03601arXiv 2504.03601
Open this paper in CoChat — free accountOpen PDF

Abstract

Training effective AI agents for multi-turn interactions requires high-quality data that captures realistic human-agent dynamics, yet such data is scarce and expensive to collect manually. We introduce APIGen-MT, a two-phase framework that generates verifiable and diverse multi-turn agent data. In the first phase, our agentic pipeline produces detailed task blueprints with ground-truth actions, leveraging a committee of LLM reviewers and iterative feedback loops. These blueprints are then transformed into complete interaction trajectories through simulated human-agent interplay. We train a family of models -- the xLAM-2-fc-r series with sizes ranging from 1B to 70B parameters. Our models outperform frontier models such as GPT-4o and Claude 3.5 on $τ$-bench and BFCL benchmarks, with the smaller models surpassing their larger counterparts, particularly in multi-turn settings, while maintaining superior consistency across multiple trials. Comprehensive experiments demonstrate that our verified blueprint-to-details approach yields high-quality training data, enabling the development of more reliable, efficient, and capable agents. We open-source 5K synthetic data trajectories and the trained xLAM-2-fc-r models to advance research in AI agents. Models at https://huggingface.co/collections/Salesforce/xlam-2-67ef5be12949d8dcdae354c4; Dataset at https://huggingface.co/datasets/Salesforce/APIGen-MT-5k and Website at https://apigen-mt.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.03601

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider.

    “Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider.”

    From $τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025) · cites this paper as [19]

    Section 3.2.2 (p. 5) describes a simulated user who reveals information while the agent uses APIs to interact with the environment; it says the user is unaware of those APIs. This supports the division of roles in the paper’s simulation, but the paper does not establish that existing conversational-agent benchmarks generally use single-control environments or that their users are merely passive information providers. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The hierarchical decomposition (Project Blueprint, Event List, Session Summaries) ensures that local interactions remain aligned with the global project narrative, mitigating fragmentation commonly observed in long-context generation.

    “This hierarchical decomposition ensures that local interactions remain aligned with the global project narrative, mitigating fragmentation commonly observed in long-context generation (Xia et al., 2025; Prabhakar et al., 2025).”

    From RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction (2026) · cites this paper as Prabhakar et al., 2025

    The Framework Overview (p. 4) says a task blueprint guides multi-turn interactions, supporting the general idea that local turns remain grounded in an overall task. But the paper describes a two-phase task-and-conversation pipeline, not a hierarchy of Project Blueprint, Event List, and Session Summaries; the cited passages also do not claim that it mitigates fragmentation in long-context generation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Less attention has been paid to the possibility of using the environment itself to constrain and shape user simulator behavior for increased reliability.

    “While most efforts have focused on introducing supervision for the user simulator, for instance, by using a generic LLM to generate or validate user responses [19], less attention has been paid to the possibility of using the environment itself to constrain and shape user simulator behavior for increased reliability, a core tenet of our approach.”

    From $τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 2

Research works printed but not linked: 50

  1. [1][1] S. Agashe, J. Han, S. Gan, J. Yang, A. Li, and X. E. Wang. Agent s: An open agentic framework that uses computers like a human. arXiv preprint arXiv:2410.08164 , 2024. - · arXiv
  2. [5][5] A. Antoniades, A. Örwall, K. Zhang, Y. Xie, A. Goyal, and W. Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285 , 2024. - · arXiv
  3. [6][6] S. Arcadinho, D. Aparício, and M. Almeida. Automated test generation to evaluate toolaugmented llms as conversational ai agents. arXiv preprint arXiv:2409.15934 , 2024. - · arXiv
  4. [7][7] D. Bahdanau, N. Gontier, G. Huang, E. Kamalloo, R. Pardinas, A. Piché, T. Scholak, O. Shliazhko, J. P. Tremblay, K. Ghanem, et al. Tapeagents: a holistic framework for agent development and optimization. arXiv preprint arXiv:2412.08445 , 2024. - · arXiv
  5. [8][8] Z. Bi, K. Han, C. Liu, Y. Tang, and Y. Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078 , 2024. - · arXiv
  6. [9][9] T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126 , 2023. - · arXiv
  7. [11][11] M. Chen, sunhaoze, T. Li, F. Yang, H. Liang, KeerLu, B. CUI, W. Zhang, Z. Zhou, and weipeng chen. Facilitating multi-turn function calling for LLMs via compositional instruction tuning. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=owP2mymrTD . -
  8. [12][12] Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.12881 , 2024. - · arXiv
  9. [13][13] Z. Chen, M. Li, Y. Huang, Y. Du, M. Fang, and T. Zhou. Atlas: Agent tuning via learning critical steps. arXiv preprint arXiv:2503.02197 , 2025. - · arXiv
  10. [15][15] T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691 . - · arXiv
  11. [16][16] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. - · arXiv
  12. [17][17] T. Ge, X. Chan, X. Wang, D. Yu, H. Mi, and D. Yu. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL https://arxiv.org/abs/2406.20094 . - · arXiv
  13. [18][18] S. A. Hayati, T. Jung, T. Bodding-Long, S. Kar, A. Sethy, J.-K. Kim, and D. Kang. Chainof-instructions: Compositional instruction tuning on large language models. arXiv preprint arXiv:2402.11532 , 2024. - · arXiv
  14. [19][19] K.-H. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments, 2025. URL https://arxiv.org/abs/2411.02305 . - · arXiv
  15. [20][20] E. Levi and I. Kadar. Intellagent: A multi-agent framework for evaluating conversational ai systems. arXiv preprint arXiv:2501.11067 , 2025. - · arXiv
  16. [21][21] G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. -
  17. [22][22] X. Li, H. Zou, and P. Liu. Torl: Scaling tool-integrated rl, 2025. URL https://arxiv.org/ abs/2503.23383 . - · arXiv
  18. [23][23] Y. Li, Y. Li, X. Wang, Y. Jiang, Z. Zhang, X. Zheng, H. Wang, H.-T. Zheng, P. Xie, P. S. Yu, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint arXiv:2411.02937 , 2024. - · arXiv
  19. [24][24] W. Liu, X. Huang, X. Zeng, X. Hao, S. Yu, D. Li, S. Wang, W. Gan, Z. Liu, Y. Yu, et al. Toolace: Winning the points of llm function calling. arXiv preprint arXiv:2409.00920 , 2024. - · arXiv
  20. [25][25] Z. Liu, J. Zhang, K. Asadi, Y. Liu, D. Zhao, S. Sabach, and R. Fakoor. Tail: Task-specific adapters for imitation learning with large pretrained models. arXiv preprint arXiv:2310.05905 , 2023. - · arXiv
  21. [26][26] Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, J. Tan, W. Yao, Z. Liu, Y. Feng, R. RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems , 37:54463-54482, 2024. -
  22. [27][27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. - · arXiv
  23. [28][28] J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, F. Bai, S. Ma, S. Ma, M. Li, G. Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682 , 2024. - · arXiv
  24. [29][29] A. Mitra, S. Patel, T. Chakrabarty, and C. Baral. Agentinstruct: An agentic framework for generating high-quality synthetic instruction data. arXiv preprint arXiv:2402.12360 , 2024. - · arXiv
  25. [30][30] J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139 , 2024. - · arXiv
  26. [31][31] J. Pan, R. Shar, J. Pfau, A. Talwalkar, H. He, and V . Chen. When benchmarks talk: Re-evaluating code llms with interactive feedback, 2025. URL https://arxiv.org/abs/2502.18413 . - · arXiv
  27. [32][32] J. S. Park, J. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology , pages 1-22, 2023. -
  28. [33][33] Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, X. Zhou, Y. Huang, C. Xiao, et al. Tool learning with foundation models. ACM Computing Surveys , 57(4):1-40, 2024. -
  29. [34][34] Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115 . - · arXiv
  30. [35][35] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '20, page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3406703. URL https://doi.org/10.1145/ 3394486.3406703 . - · DOI · Paper page
  31. [36][36] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems , 36:68539-68551, 2023. -
  32. [37][37] S. Sengupta, K. Curtis, A. Mallipeddi, A. Mathur, J. Ross, and L. Gou. Mag-v: A multi-agent framework for synthetic data generation and verification. arXiv preprint arXiv:2412.04494 , 2024. - · arXiv
  33. [38][38] J. Shim, G. Seo, C. Lim, and Y. Jo. Tooldial: Multi-turn dialogue generation method for tool-augmented language models. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=J1J5eGJsKZ . -
  34. [39][39] N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=vAElhFcKW6 . -
  35. [40][40] V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399 , 2025. - · arXiv
  36. [41][41] H. Su, R. Sun, J. Yoon, P. Yin, T. Yu, and S. Ö. Arık. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893 , 2025. - · arXiv
  37. [42][42] S. Tang, X. Pang, Z. Liu, B. Tang, R. Ye, X. Dong, Y. Wang, and S. Chen. Synthesizing post-training data for llms through multi-agent simulation. arXiv preprint arXiv:2410.14251 , 2024. - · arXiv
  38. [43][43] J. Wang, J. Zhou, M. Wen, X. Mo, H. Zhang, Q. Lin, C. Jin, X. Wang, W. Zhang, and Q. Peng. Hammerbench: Fine-grained function-calling evaluation in real mobile device scenarios. arXiv preprint arXiv:2412.16516 , 2024. - · arXiv
  39. [44][44] G. Wölflein, D. Ferber, D. Truhn, O. Arandjelovi´ c, and J. N. Kather. Llm agents making agent tools. arXiv preprint arXiv:2502.11705 , 2025. - · arXiv
  40. [46][46] J. Yang, A. Prabhakar, S. Yao, K. Pei, and K. R. Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag. In Multi-Agent Security Workshop @ NeurIPS'23 , 2023. URL https://openreview.net/forum?id=KOZwk7BFc3 . -
  41. [47][47] J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems , 36, 2024. -
  42. [48][48] R. Yang, F. Ye, J. Li, S. Yuan, Y . Zhang, Z. Tu, X. Li, and D. Yang. The lighthouse of language: Enhancing llm agents via critique-guided improvement. arXiv preprint arXiv:2503.16024 , 2025. - · arXiv
  43. [49][49] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045 , 2024. - · arXiv
  44. [50][50] F. Yin, Z. Wang, I.-H. Hsu, J. Yan, K. Jiang, Y. Chen, J. Gu, L. T. Le, K.-W. Chang, C.-Y. Lee, H. Palangi, and T. Pfister. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation, 2025. URL https://arxiv.org/abs/2503.07826 . - · arXiv
  45. [51][51] Y. Zeng, X. Ding, Y. Wang, W. Liu, W. Ning, Y. Hou, X. Huang, B. Qin, and T. Liu. Boosting tool use of large language models via iterative reinforced fine-tuning. arXiv preprint arXiv:2501.09766 , 2025. - · arXiv
  46. [52][52] J. Zhang, T. Lan, M. Zhu, Z. Liu, T. Hoang, S. Kokane, W. Yao, J. Tan, A. Prabhakar, H. Chen, et al. xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215 , 2024. - · arXiv
  47. [53][53] J. Zhang, T. Hoang, M. Zhu, Z. Liu, S. Wang, T. Awalgaonkar, A. Prabhakar, H. Chen, W. Yao, Z. Liu, et al. Actionstudio: A lightweight framework for data and training of action models. arXiv preprint arXiv:2503.22673 , 2025. - · arXiv
  48. [54][54] K. Zhang, W. Yao, Z. Liu, Y. Feng, Z. Liu, R. Rithesh, T. Lan, L. Li, R. Lou, J. Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. In The Thirteenth International Conference on Learning Representations , 2024. -
  49. [55][55] Y. Zhang, J. Lu, and N. Jaitly. Probing the multi-turn planning capabilities of LLMs via 20 question games. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1495-1516, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.82. URL https://aclanthology.org/2024.acl-long. 82/ . - · DOI
  50. [56][56] Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv.org/abs/ 2403.13372 .

Authors
Prabhakar, A., Liu, Z., Zhu, M., Zhang, J., Awalgaonkar, T., Wang, S., Liu, Z., Chen, H., Hoang, T., Niebles, J. C., Heinecke, S., Yao, W., Wang, H., Savarese, S., & Xiong, C.
Affiliations
  • Salesforce (United States) (US)
  • Emory University (US)
  • Westlake University (CN)
  • Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
72647-72669
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2504.03601
arXiv
2504.03601
OpenAlex
W4415980674
Cited by
2
References
0
Citations by year
2026: 2
FWCI
1.90
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
iterative feedback, LLM agents, synthetic data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review