CoChat
Sign Up
Cited by 4Software Engineering ResearchArtificial Intelligence ApplicationsNatural Language Processing Techniques

The DOI matches its Crossref record.

Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions,

Open access
Hasan, Mohammed Mehedi, Li, Hao, Rajbahadur, Gopi Krishnan, Adams, Bram, Hassan, Ahmed E.·2026·ACM Transactions on Software Engineering and Methodology
DOI 10.1145/3858625
Open this paper in CoChat — free accountOpen PDF

Abstract

The Model Context Protocol (MCP) introduces a standard specification that defines how Foundation Model (FM)-based agents should interact with external systems by invoking tools. However, to understand a tool’s purpose and features, FMs rely on natural-language tool descriptions, making these descriptions a critical component in guiding FMs to select the optimal tool for a given (sub)task and to pass the right arguments to the tool. While defects or smells in these descriptions can misguide FM-based agents, their prevalence and consequences in the MCP ecosystem remain unclear. Hence, we examine 856 tools spread across 103 MCP servers empirically, assess their description quality, and their impact on agent performance. We identify six components of tool descriptions from the literature, develop a scoring rubric utilizing these components, and then formalize tool description smells based on this rubric. By operationalizing this rubric through an FM-based scanner, we find that 97.1% of the analyzed tool descriptions contain at least one smell, with 56% failing to state their purpose clearly. While augmenting these descriptions for all components improves task success rates by a median of 5.85 percentage points and improves partial goal completion by 15.12%, it also increases the number of execution steps by 67.46% and regresses performance in 16.67% of cases. These results indicate that achieving performance gains is not straightforward; while execution cost can act as a trade-off, execution context can also impact. Furthermore, component ablations show that compact variants of different component combinations often preserve behavioral reliability while reducing unnecessary token overhead, enabling more efficient use of the FM context window and lower execution costs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3858625

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • By enforcing structured contracts such as JSON schemas, MCP messages, or OpenAPI-aligned calls, protocols constrain the model's generative space and make downstream execution deterministic enough to govern.

    “By enforcing structured contracts such as JSON schemas, MCP messages, or OpenAPI-aligned calls, they constrain the model's generative space and make downstream execution deterministic enough to govern [Hasan et al., 2026].”

    From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Hasan et al., 2026

    The Introduction describes MCP as exposing tools through names, descriptions, and an input schema, and §3.1 describes standardized tool discovery and invocation using JSON-RPC. Those passages support the existence of structured MCP interfaces, but the paper does not establish that JSON schemas or OpenAPI-aligned calls constrain a model’s generative space or make downstream execution deterministic enough to govern. The cited passages have no page numbers in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 117

  1. Hervé Abdi. 2010. Holm's sequential Bonferroni procedure. Encyclopedia of research design 1, 8 (2010), 1–8.
  2. Mayank Agarwal, Ibrahim Abdelaziz, Kinjal Basu, Merve Unuvar, Luis A Lastras, Yara Rizk, and Pavan Kapanipathi. 2025. Toolrm: Outcome reward models for tool-calling large language models. arXiv preprint arXiv:2509.11963 (2025). · arXiv
  3. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. 2025. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457 (2025). · arXiv
  4. Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs replace manual annotation of software engineering artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 526–538.
  5. Richard A Armstrong. 2014. When to use the B onferroni correction. Ophthalmic and physiological optics 34, 5 (2014), 502–508.
  6. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2025. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 238–255.
  7. Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David Binkley. 2012. An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 2012 28th IEEE international conference on software maintenance (ICSM). IEEE, 56–65.
  8. Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? an empirical study. Empirical Software Engineering 20, 4 (2015), 1052–1094. · DOI
  9. João Helis Bernardo, Daniel Alencar da Costa, Uirá Kulesza, and Christoph Treude. 2023. The impact of a continuous integration service on the delivery time of merged pull requests. Empirical Software Engineering 28, 4 (2023), 97. · DOI
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  11. Diego Cedrim, Alessandro Garcia, Melina Mongiovi, Rohit Gheyi, Leonardo Sousa, Rafael De Mello, Baldoino Fonseca, Márcio Ribeiro, and Alexander Chávez. 2017. Understanding the impact of refactoring on smells: A longitudinal study of 23 software projects. In Proceedings of the 2017 11th Joint Meeting on foundations of Software Engineering. 465–475.
  12. Kajohnsak Chaokromthong, Nittaya Sintao, et al. 2021. Sample size estimation using Yamane and Cochran and Krejcie and Morgan and green formulas and Cohen statistical power analysis by G* Power and comparisions. Apheit international journal of interdisciplinary social sciences and technology 10, 2 (2021), 76–86.
  13. Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, and Yanghua Xiao. 2023. Mapo: Boosting large language model performance with model-adaptive prompt optimization. In Findings of the Association for Computational Linguistics: EMNLP 2023. 3279–3304.
  14. Gaurab Chhetri, Shriyank Somvanshi, Md Monzurul Islam, Shamyo Brotee, Mahmuda Sultana Mimi, Dipti Koirala, Biplov Pandey, and Subasish Das. 2025. Model Context Protocols in Adaptive Transport Systems: A Survey. arXiv preprint arXiv:2508.19239 (2025). · arXiv
  15. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213.
  16. Jérémie F Cohen, Martin Chalumeau, Robert Cohen, Daniël A Korevaar, Babak Khoshnood, and Patrick MM Bossuyt. 2015. Cochran's Q test was useful to assess heterogeneity in likelihood ratios in studies of diagnostic accuracy. Journal of clinical epidemiology 68, 3 (2015), 299–306. · DOI
  17. Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A Malin, and Sricharan Kumar. 2025. Automatic prompt optimization via heuristic search: A survey. arXiv preprint arXiv:2502.18746 (2025). · arXiv
  18. Jéssica Soares Dos Santos Leonardo Guerreiro Azevedo Elton FS Soares Raphael Melo Thiago and Viviane Torres da Silva. 2020. Analysis of Tools for REST Contract Specification in Swagger/OpenAPI.. In ICEIS (2). 201–208. · DOI
  19. Abul Ehtesham, Aditi Singh, and Saket Kumar. 2025. Enhancing Clinical Decision Support and EHR Insights through LLMs and the Model Context Protocol: An Open-Source MCP-FHIR Framework. arXiv preprint arXiv:2506.13800 (2025). · arXiv
  20. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations. 150–158.
  21. Tosin Fadahunsi, Giordano d’Aloisio, Antinisca Di Marco, and Federica Sarro. 2025. How do generative models draw a software engineer? A case study on stable diffusion bias. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering-Companion (SANER-C). IEEE, 9–16.
  22. Shiqing Fan, Xichen Ding, Liang Zhang, and Linjian Mo. 2025. Mcptoolbench++: A large scale ai agent model context protocol mcp tool use benchmark. arXiv preprint arXiv:2508.07575 (2025). · arXiv
  23. Xiang Fei, Xiawu Zheng, and Hao Feng. 2025. Mcp-zero: Active tool discovery for autonomous llm agents. arXiv preprint arXiv:2506.01056 (2025). · arXiv
  24. Henning Femmer, Daniel Méndez Fernández, Stefan Wagner, and Sebastian Eder. 2017. Rapid quality assurance with requirements smells. Journal of Systems and Software 123 (2017), 190–213. · DOI
  25. Francesca Arcelli Fontana, Ilaria Pigazzini, Riccardo Roveda, Damian Tamburri, Marco Zanoni, and Elisabetta Di Nitto. 2017. Arcan: A tool for architectural smells detection. In 2017 IEEE International Conference on Software Architecture Workshops (ICSAW). IEEE, 282–285.
  26. Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley Professional.
  27. Yi Gao, Xing Hu, Xiaohu Yang, and Xin Xia. 2025. Automated Unit Test Refactoring. Proceedings of the ACM on Software Engineering 2, FSE (2025), 713–733.
  28. Joshua Garcia, Daniel Popescu, George Edwards, and Nenad Medvidovic. 2009. Identifying architectural bad smells. In 2009 13th European Conference on Software Maintenance and Reengineering. IEEE, 255–258.
  29. Joshua Garcia, Daniel Popescu, George Edwards, and Nenad Medvidovic. 2009. Toward a catalogue of architectural bad smells. In International conference on the quality of software architectures. Springer, 146–162.
  30. Isha Hameed, Samuel Sharpe, Daniel Barcklow, Justin Au-Yeung, Sahil Verma, Jocelyn Huang, Brian Barr, and C Bayan Bruss. 2022. BASED-XAI: Breaking ablation studies down for explainable artificial intelligence. arXiv preprint arXiv:2207.05566 (2022). · arXiv
  31. Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2025. An empirical study of testing practices in open source AI agent frameworks and agentic applications. arXiv preprint arXiv:2509.19185 (2025). · arXiv
  32. Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2025. Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers. arXiv preprint arXiv:2506.13538 (2025). · arXiv
  33. Mohammad Mehedi Hassan and Akond Rahman. 2022. As code testing: Characterizing test quality in open source ansible development. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST).
  34. Erik Henriksson, Otto Tarkka, and Filip Ginter. 2025. FinerWeb-10BT: Refining web data with LLM-based line-level filtering. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). 258–268.
  35. Thomas Hirsch and Birgit Hofer. 2022. A systematic literature review on benchmarks for evaluating debugging approaches. Journal of Systems and Software 192 (2022), 111423. · DOI
  36. David C Hoaglin. 2016. Misunderstandings about Q and ‘Cochran's Q test’in meta-analysis. Statistics in medicine 35, 4 (2016), 485–495.
  37. Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv preprint arXiv:2503.23278 (2025). · arXiv
  38. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.00675 (2023). · arXiv
  39. Marcel Jerzyk and Lech Madeyski. 2023. Code smells: A comprehensive online catalog and taxonomy. In Developments in Information and Knowledge Management Systems for Business Applications: Volume 7. Springer, 543–576.
  40. Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology 7, 4 (2015), 396.
  41. Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002. CCFinder: A multilinguistic token-based code clone detection system for large scale source code. IEEE transactions on software engineering 28, 7 (2002), 654–670.
  42. Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. Ai agents that matter. arXiv preprint arXiv:2407.01502 (2024). · arXiv
  43. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714 (2023). · arXiv
  44. Foutse Khomh, Massimiliano Di Penta, and Yann-Gael Gueheneuc. 2009. An exploratory study of the impact of code smells on software change-proneness. In 2009 16th Working Conference on Reverse Engineering. IEEE, 75–84.
  45. Foutse Khomh, Stéphane Vaucher, Yann-Gaël Guéhéneuc, and Houari Sahraoui. 2009. A bayesian approach for the detection of code and design smells. In 2009 Ninth International Conference on Quality Software. IEEE, 305–314.
  46. Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine 15, 2 (2016), 155–163. · DOI · Paper page
  47. Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta, and Yann Gaël Guéhéneuc. 2020. Code smells and refactoring: A tertiary systematic review of challenges and observations. Journal of Systems and Software 167 (2020), 110610. · DOI
  48. Jiahui Liang, Weiqin Zou, Jingxuan Zhang, Zhiqiu Huang, and Chenxing Sun. 2021. A deep method renaming prediction and refinement approach for Java projects. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). IEEE, 404–413.
  49. Yao-Yang Liu, Zhen Zheng, Feng Zhang, Jin-Cheng Feng, Yi-Yang Fu, Ji-Dong Zhai, Bing-Sheng He, Xiao Zhang, and Xiao-Yong Du. 2026. A comprehensive taxonomy of prompt engineering techniques for large language models. Frontiers of Computer Science 20, 3 (2026), 2003601. · DOI
  50. Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. 2024. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems 37 (2024), 54463–54482.
  51. Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, et al. 2025. Mcpeval: Automatic mcp-based deep evaluation for ai agent models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 373–402.
  52. Keila Lucas, Rohit Gheyi, Elvys Soares, Márcio Ribeiro, and Ivan Machado. 2024. Evaluating large language models in detecting test smells. arXiv preprint arXiv:2407.19261 (2024). · arXiv
  53. Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. 2025. Mcp-universe: Benchmarking large language models with real-world model context protocol servers. arXiv preprint arXiv:2508.14704 (2025). · arXiv
  54. Guillermo Macbeth, Eugenia Razumiejczyk, and Rubén Daniel Ledesma. 2011. Cliff's Delta Calculator: A non-parametric effect size program for two groups of observations. Universitas Psychologica 10, 2 (2011), 545–555. · DOI · Paper page
  55. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60.
  56. Mika Mantyla, Jari Vanhanen, and Casper Lassenius. 2003. A taxonomy and an initial empirical study of bad smells in code. In International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings. IEEE, 381–384.
  57. Raúl Marticorena, Carlos López, and Yania Crespo. 2006. Extending a taxonomy of bad code smells with metrics. In Proceedings of 7th International Workshop on Object-Oriented Reengineering (WOOR). Citeseer, 6.
  58. Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, et al. 2025. A survey of context engineering for large language models. arXiv preprint arXiv:2507.13334 (2025). · arXiv
  59. Guozhao Mo, Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. 2025. Livemcpbench: Can agents navigate an ocean of mcp tools? arXiv preprint arXiv:2508.01780 (2025). · arXiv
  60. Naouel Moha, Yann-Gaël Guéhéneuc, Laurence Duchien, and Anne-Francoise Le Meur. 2009. Decor: A method for the specification and detection of code and design smells. IEEE Transactions on Software Engineering 36, 1 (2009), 20–36. · DOI
  61. Michael Mohan, Des Greer, and Paul McMullan. 2016. Technical debt reduction using search based automated refactoring. Journal of Systems and Software 120 (2016), 183–194. · DOI
  62. Purnima Naik, Salomi Nelaballi, Venkata Sai Pusuluri, and Dae-Kyoo Kim. 2024. Deep learning-based code refactoring: A review of current knowledge. Journal of Computer Information Systems 64, 2 (2024), 314–328. · DOI
  63. Steffen Olbrich, Daniela S Cruzes, Victor Basili, and Nico Zazworka. 2009. The evolution and impact of code smells: A case study of two open source systems. In 2009 3rd international symposium on empirical software engineering and measurement. IEEE, 390–400.
  64. Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695 (2024). · arXiv
  65. Wendkûuni C Ouédraogo, Yinghua Li, Kader Kaboré, Xunzhu Tang, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Test smells in LLM-Generated Unit Tests. arXiv preprint arXiv:2410.10628 (2024). · arXiv
  66. Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Denys Poshyvanyk, and Andrea De Lucia. 2014. Mining version histories for detecting code smells. IEEE Transactions on Software Engineering 41, 5 (2014), 462–489. · DOI
  67. Gabriele Pandini, Antonio Martini, Adela Nedisan Videsjorden, and Francesca Arcelli Fontana. 2025. An exploratory study on architectural smell refactoring using Large Languages Models. In 2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C). IEEE, 462–471.
  68. Michael J Parker, Caitlin Anderson, Claire Stone, and YeaRim Oh. 2024. A large language model approach to educational survey feedback analysis. International journal of artificial intelligence in education (2024), 1–38.
  69. Aditya Pathak, Rachit Gandhi, Vaibhav Uttam, Arnav Ramamoorthy, Pratyush Ghosh, Aaryan Raj Jindal, Shreyash Verma, Aditya Mittal, Aashna Ased, Chirag Khatri, et al. 2025. Rubric is all you need: Improving llm-based code evaluation with question-specific rubrics. In Proceedings of the 2025 ACM Conference on International Computing Education Research V. 1. 181–195.
  70. Debalina Ghosh Paul, Hong Zhu, and Ian Bayley. 2025. Investigating The Smells of LLM Generated Code. arXiv preprint arXiv:2510.03029 (2025). · arXiv
  71. Anthropic PBC. 2025. Agent Skills: What are skills. https://agentskills.io/what-are-skills Accessed: 2026-01-18.
  72. Matilda QR Pembury Smith and Graeme D Ruxton. 2020. Effective use of the McNemar test. Behavioral Ecology and Sociobiology 74, 11 (2020), 133. · DOI
  73. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023). · arXiv
  74. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025. Tool learning with large language models: A survey. Frontiers of Computer Science 19, 8 (2025), 198343. · DOI
  75. Krishna Ronanki, Beatriz Cabrero-Daniel, and Christian Berger. 2024. Prompt smells: An omen for undesirable generative AI outputs. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 286–287.
  76. Graeme D Ruxton and Guy Beauchamp. 2008. Time for some a priori thinking about post hoc testing. Behavioral ecology 19, 3 (2008), 690–693.
  77. Ahmed R Sadik and Siddhata Govind. 2025. Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3. arXiv preprint arXiv:2504.16027 (2025). · arXiv
  78. Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927 (2024). · arXiv
  79. Anjana Sarkar and Soumyendu Sarkar. 2025. Survey of LLM Agent Communication with MCP: A Software Design Pattern Centric Review. arXiv preprint arXiv:2506.05364 (2025). · arXiv
  80. Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. 2025. Prompt Injection Attack to Tool Selection in LLM Agents. arXiv preprint arXiv:2504.19793 (2025). · arXiv
  81. Patrick E Shrout and Joseph L Fleiss. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin 86, 2 (1979), 420.
  82. Mohammed Latif Siddiq, Lindsay Roney, Jiahao Zhang, and Joanna Cecilia Da Silva Santos. 2024. Quality assessment of chatgpt generated code and their use by developers. In Proceedings of the 21st international conference on mining software repositories. 152–156.
  83. Dag IK Sjøberg, Aiko Yamashita, Bente CD Anda, Audris Mockus, and Tore Dybå. 2012. Quantifying the effect of code smells on maintenance effort. IEEE Transactions on Software Engineering 39, 8 (2012), 1144–1156. · DOI · Paper page
  84. Ion Stoica, Matei Zaharia, Joseph Gonzalez, Ken Goldberg, Koushik Sen, Hao Zhang, Anastasios Angelopoulos, Shishir G Patil, Lingjiao Chen, Wei-Lin Chiang, et al. 2024. Specifications: The missing link to making the development of llm systems an engineering discipline. arXiv preprint arXiv:2412.05299 (2024). · arXiv
  85. Balázs Szalontai, András Vadász, Zsolt Richárd Borsi, Teréz A Várkonyi, Balázs Pintér, and Tibor Gregorics. 2021. Detecting and fixing nonidiomatic snippets in python source code with deep learning. In Proceedings of SAI Intelligent Systems Conference. Springer, 129–147.
  86. Aditi Tiwari, Akshit Bhalla, and Darshan Prasad. 2025. Model Context Protocol for Vision Systems: Audit, Security, and Protocol Extensions. arXiv preprint arXiv:2509.22814 (2025). · arXiv
  87. Nikolaos Tsantalis and Alexander Chatzigeorgiou. 2009. Identification of move method refactoring opportunities. IEEE Transactions on Software Engineering 35, 3 (2009), 347–367. · DOI
  88. Nikolaos Tsantalis and Alexander Chatzigeorgiou. 2011. Identification of extract method refactoring opportunities for the decomposition of methods. Journal of Systems and Software 84, 10 (2011), 1757–1782. · DOI
  89. Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An empirical investigation into the nature of test smells. In Proceedings of the 31st IEEE/ACM international conference on automated software engineering. 4–15.
  90. Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk. 2019. On learning meaningful code changes via neural machine translation. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 25–36.
  91. Roberto Verdecchia, Giuseppe Procaccianti, Patricia Lago, et al. 2018. Empirical evaluation of the energy impact of refactoring code smells. In 5th International Conference on Information and Communication Technology for Sustainability. ICT4S2018. EasyChair, 365–383.
  92. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796 (2024). · arXiv
  93. Tijana Vislavski, Gordana Rakić, Nicolás Cardozo, and Zoran Budimac. 2018. LICCA: A tool for cross-language clone detection. In 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER). IEEE, 512–516.
  94. Antonio Vitale, Rocco Oliveto, and Simone Scalabrino. 2025. A catalog of data smells for coding tasks. ACM Transactions on Software Engineering and Methodology 34, 4 (2025), 1–32. · DOI
  95. Andreas Vogelsang, Alexander Korn, Giovanna Broccia, Alessio Ferrari, Jannik Fischbach, and Chetan Arora. 2025. On the impact of requirements smells in prompts: The case of automated traceability. In 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, 51–55.
  96. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 9440–9450.
  97. Pengcheng Wang, Jeffrey Svajlenko, Yanzhao Wu, Yun Xu, and Chanchal K Roy. 2018. CCAligner: a token based large-gap clone detector. In Proceedings of the 40th International Conference on Software Engineering. 1066–1077.
  98. Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977.
  99. Zihan Wang, Rui Zhang, Yu Liu, Wenshu Fan, Wenbo Jiang, Qingchuan Zhao, Hongwei Li, and Guowen Xu. 2025. Mpma: Preference manipulation attack against model context protocol. arXiv preprint arXiv:2505.11154 (2025). · arXiv
  100. Florensia Widjaja, Zhangtianyi Chen, and Juexiao Zhou. 2025. BioinfoMCP: A Unified Platform Enabling MCP Interfaces in Agentic Bioinformatics. arXiv preprint arXiv:2510.02139 (2025). · arXiv
  101. David M Williamson, Xiaoming Xi, and F Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational measurement: issues and practice 31, 1 (2012), 2–13.
  102. Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials (2007), 1–3.
  103. Di Wu, Fangwen Mu, Lin Shi, Zhaoqiang Guo, Kui Liu, Weiguang Zhuang, Yuqi Zhong, and Li Zhang. 2024. ismell: Assembling llms with expert toolsets for code smell detection and refactoring. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1345–1357.
  104. Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024. Prompt optimization with EASE? efficient ordering-aware automated selection of exemplars. Advances in Neural Information Processing Systems 37 (2024), 122706–122740.
  105. Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. 2025. A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint arXiv:2508.05668 (2025). · arXiv
  106. Weikai Xu, Chengrui Huang, Shen Gao, and Shuo Shang. 2025. LLM-Based Agents for Tool Learning: A Survey: W. Xu et al. Data Science and Engineering (2025), 1–31.
  107. Aiko Yamashita and Leon Moonen. 2012. Do code smells reflect important maintainability aspects?. In 2012 28th IEEE international conference on software maintenance (ICSM). IEEE, 306–315.
  108. Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, et al. 2025. MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents. arXiv preprint arXiv:2506.07672 (2025). · arXiv
  109. Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kästner, and Tongshuang Wu. 2025. What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts. arXiv preprint arXiv:2505.13360 (2025). · arXiv
  110. Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Jason Wu. 2023. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. arXiv preprint arXiv:2306.01150 (2023). · arXiv
  111. Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong, Mian Zhang, Yebowen Hu, Shujian Liu, Simin Ma, Song Wang, et al. 2025. Livemcp-101: Stress testing and diagnosing mcp-enabled agents on challenging queries. arXiv preprint arXiv:2508.15760 (2025). · arXiv
  112. Beiqi Zhang, Peng Liang, Qiong Feng, Yujia Fu, and Zengyang Li. 2024. Copilot-in-the-loop: Fixing code smells in copilot-generated python code using copilot. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2230–2234.
  113. Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, et al. 2025. From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents. arXiv preprint arXiv:2506.18959 (2025). · arXiv
  114. Zhimin Zhao, Abdul Ali Bangash, Filipe Roseiro Côgo, Bram Adams, and Ahmed E Hassan. 2025. On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards. IEEE Transactions on Software Engineering (2025).
  115. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797 (2023). · arXiv
  116. Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. 2025. ComplexFuncBench: exploring multi-step and constrained function calling under long-context scenario. arXiv preprint arXiv:2501.10132 (2025). · arXiv
  117. Minhaz Fahim Zibran. 2007. Chi-squared test of independence. Department of Computer Science, University of Calgary, Alberta, Canada 1, 1 (2007), 1–7.

Cited by

Sorted by citations

Metadata

Authors
Hasan, M. M., Li, H., Rajbahadur, G. K., Adams, B., & Hassan, A. E.
Affiliations
Queen's University (CA)
ORCID
Venue
ACM Transactions on Software Engineering and Methodology
Publisher
Association for Computing Machinery (ACM)
Published
2026-10-15
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
DOI
10.1145/3858625
OpenAlex
W7129522927
Cited by
4
References
0
Citations by year
2026: 4
FWCI
18.18
Type
journal article
Language
en
Open access
Licence
cc-by-nc-sa
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
Model Context Protocol, tool selection, tool invocation, execution cost, token efficiency, context window, agent performance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review