Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Bots are software systems designed to support users by automating specific processes, tasks, or activities. When these systems implement a conversational component to interact with users, they are also known as conversational agents or chatbots . Bots—particularly in their conversation-oriented version and AI-powered—have seen increased adoption over time for software development and engineering purposes. Despite their exciting potential, which has been further enhanced by the advent of Generative AI and Large Language Models, bots still face challenges in terms of development and integration into the development cycle, as practitioners report that bots can add difficulties rather than provide improvements. In this work, we aim to provide a taxonomy for characterizing bots, as well as a series of challenges for their adoption in software engineering, accompanied by potential mitigation strategies. To achieve our objectives, we conducted a multivocal literature review , examining both research and practitioner literature. Through such an approach, we hope to contribute to both researchers and practitioners by providing (i) a series of future research directions to pursue, (ii) a list of strategies to adopt for improving the use of bots for software engineering purposes, and (iii) fostering technology and knowledge transfer from the research field to practice—one of the primary goals of multivocal literature reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Uncanny Valley [From the Field]
Mori, M., MacDorman, K., & Kageki, N. (2012) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2012.2192811 · Cited by 3,542
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · cited 1×
“The following subsections report each identified challenge in detail.”
Chatbots applications in education: A systematic review
Okonkwo, C. W., & Ade-Ibijola, A. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100033 · Cited by 879
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Searching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814 · cited 1×
“Regarding the second question, since the topic of bots in SE is young, there is not enough literature to determine the presence of a lack of consensus.”
Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method
Ogawa, R. T., & Malen, B. (1991) · Review of Educational Research · doi:10.3102/00346543061003265 · Cited by 247 · cited 3×
“The Search Phase consists of two sub-phases.”
When and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182 · cited 6×
“(“bot(s)” OR “chatbot(s)” OR “chat bot(s)” OR “conversational agent(s)”) AND (“software engineering”) As for the GL, following the guidelines provided by [122, 133], we started from a general search string to define a more specific one based on the goal of each research question.”
InfluentialTowards Human-Bot Collaborative Software Architecting with ChatGPT
Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M. S., & Mikkonen, T. (2023) · Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3593434.3593468 · Cited by 168 · cited 10×
“Other studies propose bots to improve and speed up agile processes by automatizing the agile retrospective step by collecting metrics from different sources [A7, A46].”
Influential(2016) · doi:10.1145/2950290.2983989 · Cited by 153
Research works printed but not linked: 115
- Mairieli Wessel Igor Wiese Igor Steinmacher and Marco Aurelio Gerosa. 2021. Don’t disturb me: Challenges of interacting with software bots on open source software projects. Proceedings of the ACM on Human-Computer Interaction 5 CSCW2 (2021) 1–21. · DOI
- Shayan Zamanirad Boualem Benatallah Moshe Chai Barukh Fabio Casati and Carlos Rodriguez. 2017. Programming bots by synthesizing natural language expressions into api invocations. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE’17) 832–837. · DOI
- Saurabh Srivastava and T. V. Prabhakar. 2019. A reference architecture for applications with conversational components. In Proceedings of the IEEE 10th International Conference on Software Engineering and Service Science (ICSESS’19). IEEE Los Alamitos CA 1–5. · DOI
- Rijnard van Tonder and Claire Le Goues. 2019. Towards s/engineer/bot: Principles for program repair bots. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). IEEE Los Alamitos CA. 43–47. · DOI
- Martin Monperrus. 2019. Explainable software bot contributions: Case study of automated bug fixes. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). 12–15 · DOI
- Fabian Gilson and Danny Weyns. 2019. When natural language processing jumps into collaborative software engineering. In Proceedings of the IEEE International Conference on Software Architecture Companion (ICSA-C’19). 238–241. · DOI
- Ranci Ren John W. Castro Adrián Santos Sara Pérez-Soler Silvia T. Acuña and Juan de Lara. 2020. Collaborative modelling: Chatbots or on-line tools? an experimental study. In Proceedings of the Evaluation and Assessment in Software Engineering (EASE ’20). Association for Computing Machinery New York NY. 260–269. · DOI
- Fabian Gilson Sam Annand and Jack Steel. 2020. Recording software design decisions on the fly. In Joint Proceedings of the 3rd Software Engineering Education Workshop (SEED 2020) and the Workshop on NLP Advancements for Software Engineering at the Asia-Pacific Software Engineering Conference (SEED/NLPaSE@ APSEC’20). 53–66.
- Justine Winata Purwoko Tegar Abdullah Budiman Wijaya Alexander Agung Santoso Gunawan and Karen Etania Saputra. 2023. Analysis chatgpt potential: Transforming software development with ai chat bots. In Proceedings of the International Conference on Networking Electrical Engineering Computer Science and Technology (IConNECT’23). IEEE Los Alamitos CA. 36–41. · DOI
- Rijul Saini Gunter Mussbacher Jin L. C. Guo and Jörg Kienzle. 2022. Automated interactive and traceable domain modelling empowered by artificial intelligence. Softw. Syst. Model. 21 3 (2022) 1015–1045. · DOI
- Lissette Almonte Sara Pérez-Soler Esther Guerra Iván Cantador and Juan de Lara. Automating the synthesis of recommender systems for modelling languages. In Proceedings of the 14th ACM SIGPLAN International Conference on Software Language Engineering pages 22–35 2021. · doi:10.1145/3486608.3486905
- Ilham Qasse Shailesh Mishra Björn þór Jónsson Foutse Khomh and Mohammad Hamdaqa. 2023. Chat2code: A chatbot for model specification and code generation the case of smart contracts. In Proceedings of the IEEE International Conference on Software Services Engineering (SSE’23). IEEE Los Alamitos CA. 50–60. · DOI
- Gian Luca Scoccia. 2023. Exploring early adopters’ perceptions of chatgpt as a code generation tool. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW’23). IEEE Los Alamitos CA. 88–93. · DOI
- Sara Pérez-Soler Esther Guerra and Juan de Lara. 2019. Flexible modelling using conversational agents. In Proceedings of the ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C’19). 478–482. · doi:10.1109/models-c.2019.00076
- Marvin Wyrich and Justus Bogner. 2019. Towards an autonomous bot for automatic source code refactoring. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). IEEE Los Alamitos CA. 24–28. · DOI
- Vahid Alizadeh Mohamed Amine Ouali Marouane Kessentini and Meriem Chater. 2019. Refbot: intelligent software refactoring bot. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE’19). IEEE Los Alamitos CA. 823–834. · DOI
- Zhewei Hu and Edward F. Gehringer. 2019. Improving feedback on github pull requests: A bots approach. In Proceedings of the IEEE Frontiers in Education Conference (FIE’19). 1–9. · doi:10.1109/fie43999.2019.9028685
- Antônio Carvalho Welder Luz Diego Marcílio Rodrigo Bonifácio Gustavo Pinto and Edna Dias Canedo. 2020. C-3pr: A bot for fixing static analysis violations via pull requests. In Proceedings of the IEEE 27th International Conference on Software Analysis Evolution and Reengineering (SANER’20). 161–171. · DOI
- Marvin Wyrich. Regina Hebig. Stefan Wagner. and Riccardo Scandariato. 2020. Perception and acceptance of an autonomous refactoring bot. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence—Volume 1 (ICAART’20). INSTICC SciTePress. 303–310. · doi:10.5220/0009168803030310
- Marvin Wyrich Raoul Ghit Tobias Haller and Christian Müller. 2021. Bots don’t mind waiting do they? Comparing the interaction with automatically and manually created pull requests. arXiv:2103.03591. Retrieved from https://arxiv.org/abs/2103.03591 · DOI
- Dragos Serban Bart Golsteijn Ralph Holdorp and Alexander Serebrenik. 2021. Saw-bot: Proposing fixes for static analysis warnings with github suggestions. In Proceedings of the IEEE/ACM 3rd International Workshop on Bots in Software Engineering (BotSE’21). 26–30. · DOI
- Margaret-Anne Storey and Alexey Zagalsky. 2016. Disrupting developer productivity one bot at a time. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering(FSE’16) Association for Computing Machinery New York NY. 928–931.
- Simon Urli Zhongxing Yu Lionel Seinturier and Martin Monperrus. 2018. How to design a program repair bot? insights from the repairnator project. In Proceedings of the IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP’18). IEEE Los Alamitos CA. 95–104. · DOI
- Elahe Paikari and André van der Hoek. 2018. A framework for understanding chatbots and their future. In Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE ’18) Association for Computing Machinery New York NY. 13–16 · DOI · Paper page
- Linda Erlenhov Francisco Gomes de Oliveira Neto and Philipp Leitner. 2020. An empirical study of bots in software development: Characteristics and challenges from a practitioner’s perspective. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20). Association for Computing Machinery New York NY. 445–455. · DOI
- Linda Erlenhov Francisco Gomes de Oliveira Neto Martin Chukaleski and Samer Daknache. 2020. Challenges and guidelines on designing test cases for test bots. 2020. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW’20). Association for Computing Machinery New York NY. 41–45. · DOI
- Dušan Okanović Samuel Beck Lasse Merz Christoph Zorn Leonel Merino André van Hoorn and Fabian Beck. 2020. Can a chatbot support software engineers with load testing? approach and experiences. In Proceedings of the ACM/SPEC International Conference on Performance Engineering(ICPE ’20) Association for Computing Machinery New York NY. 120–129. · DOI
- Tapajit Dey Sara Mousavi Eduardo Ponce Tanner Fry Bogdan Vasilescu Anna Filippova and Audris Mockus. Detecting and characterizing bots that commit code. 2020. In Proceedings of the 17th International Conference on Mining Software Repositories (MSR ’20) page 209–219 Association for Computing Machinery New York NY. 209–219. · DOI · Paper page
- Tapajit Dey Bogdan Vasilescu and Audris Mockus. 2020. An exploratory study of bot commits. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops(ICSEW’20). Association for Computing Machinery New York NY. 61–65.
- Runzhi He Hao He Yuxia Zhang and Minghui Zhou. 2023. Automating dependency updates in practice: An exploratory study on github dependabot. IEEE Transactions on Software Engineering 49 8 (2023) 4004–4022. · DOI
- Ivan Beschastnikh Mircea F. Lungu and Yanyan Zhuang. Accelerating software engineering research adoption with analysis bots. In 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER) pages 35–38 2017. · DOI
- Mairieli Wessel Igor Steinmacher Igor Wiese and Marco A Gerosa. 2019. Should i stale or should i close? an analysis of a bot that closes abandoned issues and pull requests. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). IEEE Los Alamitos CA. 38–42. · DOI
- Ehsan Mirsaeedi and Peter C. Rigby. 2020. Mitigating turnover with code review recommendation: Balancing expertise workload and knowledge distribution. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering(ICSE ’20) Association for Computing Machinery New York NY. 1183–1195 · DOI · Paper page
- Gianmario Voria Viviana Pentangelo Antonio Della Porta Stefano Lambiase Gemma Catolino Fabio Palomba and Filomena Ferrucci. 2022. Community smell detection and refactoring in slack: The cadocs project. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’22). IEEE Los Alamitos CA. 469–473. · DOI
- Zhendong Wang Yi Wang and David Redmiles. 2023. Optimizing workflow for elite developers: Perspectives on leveraging se bots. In Proceedings of the IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE’23). IEEE Los Alamitos CA. 23–27. · DOI
- Abdelrahman H. Hefny Georgios A. Dafoulas and Manal A. Ismail. 2021. A proactive management assistant chatbot for software engineering teams: Prototype and preliminary evaluation. In Proceedings of the 3rd Novel Intelligent and Leading Emerging Sciences Conference (NILES’21). IEEE Los Alamitos CA. 295–300 · DOI
- Soomin Kim Jinsu Eun Changhoon Oh Bongwon Suh and Joonhwan Lee. 2020. Bot in the bunch: Facilitating group chat discussion by improving efficiency and participation with a chatbot. In Proceedings of the CHI Conference on Human Factors in Computing Systems(CHI’20). Association for Computing Machinery New York NY. 1–13. · DOI · Paper page
- Elahe Paikari JaeEun Choi SeonKyu Kim Sooyoung Baek MyeongSoo Kim SeungEon Lee ChaeYeon Han YoungJae Kim KaHye Ahn Chan Cheong et al. 2019. A chatbot for conflict detection and resolution. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19) pages 29–33. IEEE Los Alamitos CA. 29–33. · DOI
- Jhonny Cerezo Juraj Kubelka Romain Robbes and Alexandre Bergel. 2019. Building an expert recommender chatbot. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). 59–63. · DOI
- Arkadip Basu and Kunal Banerjee. 2021. Designing a bot for efficient distribution of service requests. In Proceedings of the IEEE/ACM Third International Workshop on Bots in Software Engineering (BotSE’21). 16–20. · doi:10.1109/botse52550.2021.00011
- Zeyu Ni Beijun Shen Yuting Chen Zhangyuan Meng and Junming Cao. 2019. Crowdevbot: A task-oriented conversational bot for software crowdsourcing platform (s). In Proceedings of the International Conference on Software Engineering and Knowledge Engineering (SEKE’19) 410–522. · DOI
- Mairieli Wessel Alexander Serebrenik Igor Wiese Igor Steinmacher and Marco A. Gerosa. 2020. Effects of adopting code review bots on pull requests to oss projects. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’20) 1–11. · DOI
- Mairieli Wessel Ahmad Abdellatif Igor Wiese Tayana Conte Emad Shihab Marco A. Gerosa and Igor Steinmacher. 2022. Bots for pull requests: The good the bad and the promising. In Proceedings of the 44th International Conference on Software Engineering. 274–286. · DOI
- Mairieli Wessel Alexander Serebrenik Igor Wiese Igor Steinmacher and Marco A. Gerosa. Quality gatekeepers: investigating the effects of code review bots on pull request activities. Emp. Softw. Eng. 27(5) 108 (2022). · DOI
- Eric Ribeiro Ronan Nascimento Igor Steinmacher Laerte Xavier Marco Gerosa Hugo de Paula and Mairieli Wessel. 2022. Together or apart? investigating a mediator bot to aggregate bot’s comments on pull requests. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’22). IEEE Los Alamitos CA. 434–438. · DOI
- Christoph Matthies Franziska Dobrigkeit and Guenter Hesse. 2019. An additional set of (automated) eyes: Chatbots for agile retrospectives. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). IEEE Los Alamitos CA. 34–37. · DOI
- Yuxia Zhang Zhiqing Qiu Klaas-Jan Stol Wenhui Zhu Jiaxin Zhu Yingchen Tian and Hui Liu. 2024. Automatic commit message generation: A critical review and directions for future work. IEEE Trans. Softw. Eng. · DOI · Paper page
- Rahul Kumar Chetan Bansal Chandra Maddila Nitin Sharma Shawn Martelock and Ravi Bhargava. 2019. Building sankie: An ai platform for devops. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE). 48–53. · DOI
- Chun-Ting Lin Shang-Pin Ma and Yu-Wen Huang. 2020. Msabot: A chatbot framework for assisting in the development and operation of microservice-based systems. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops(ICSEW’20) Association for Computing Machinery New York NY. 36–40. · DOI
- Scott A. Carr Francesco Logozzo and Mathias Payer. 2016. Automatic contract insertion with ccbot. IEEE Trans. Softw. Eng. 43 8 (2016) 701–714. · DOI
- Yuan Tian Ferdian Thung Abhishek Sharma and David Lo. 2017. Apibot: Question answering bot for api documentation. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE’17) 153–158. · DOI
- Katsunori Fukui Tomoki Miyazaki and Masao Ohira. 2019. Suggesting questions that match each user’s expertise in community question and answering services. In Proceedings of the 20th IEEE/ACIS International Conference on Software Engineering Artificial Intelligence Networking and Parallel/Distributed Computing (SNPD’19). 501–506. · doi:10.1109/snpd.2019.8935747
- Chris Brown and Chris Parnin. 2019. Sorry to bother you: Designing bots for effective recommendations. In Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE’19). 54–58. · DOI
- Chaiyakarn Khanan Worawit Luewichana Krissakorn Pruktharathikoon Jirayus Jiarpakdee Chakkrit Tantithamthavorn Morakot Choetkiertikul Chaiyong Ragkhitwetsagul and Thanwadee Sunetnanta. 2020. Jitbot: an explainable just-in-time defect prediction bot. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 1336–1339. · DOI
- Ricardo Romero Esteban Parra and Sonia Haiduc. 2020. Experiences building an answer bot for gitter. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops(ICSEW’20). Association for Computing Machinery New York NY. 66–70. · DOI
- James Dominic Charles Ritter and Paige Rodeghero. 2020. Onboarding bot for newcomers to software engineering. In Proceedings of the International Conference on Software and System Processes(ICSSP ’20). Association for Computing Machinery New York NY. 91–94. · DOI
- Preetha Chatterjee Kostadin Damevski and Lori Pollock. 2021. Automatic extraction of opinion-based q&a from online developer chats. In Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering (ICSE’21). IEEE Los Alamitos CA. 1260–1272. · DOI
- Mahsa H. Sadi and Eric Yu. 2021. Rapid: a knowledge-based assistant for designing web apis. Requir. Eng. 26 2 (2021) 185–236. · DOI
- Peter Robe and Sandeep Kaur Kuttal. 2022. Designing pairbuddy—A conversational agent for pair programming. ACM Trans. Comput.-Hum. Interact. 29 4 (2022) 1–44. · DOI · Paper page
- Peter Robe Sandeep K. Kuttal Jake AuBuchon and Jacob Hart. 2022. Pair programming conversations with agents vs. developers: challenges and opportunities for se community. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 319–331. · DOI
- Carlene Lebeuf Margaret-Anne Storey and Alexey Zagalsky. 2018. Software bots. IEEE Softw. 35 1 (2018) 18–23 January 2018. · DOI · Paper page
- Samaneh Saadat Natalia Colmenares and Gita Sukthankar. 2021. Do bots modify the workflow of github teams? In Proceedings of the IEEE/ACM 3rd International Workshop on Bots in Software Engineering (BotSE’21). 1–5. · DOI
- André M. Pinheiro Caio S. Rabello Leonardo B. Furtado Gustavo Pinto and Cleidson R. B. de Souza. 2019. Expecting the unexpected: Distilling bot development challenges and motivations. In Proceedings of the IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE’19). 51–52. · DOI
- Jinho Lee Inseok Hwang Thomas S. Hubregtsen Anne E. Gattiker and Christopher M. Durham. 2019. Accelerating conversational agents built with off-the-shelf modularized services. IEEE Perv. Comput. 18 2 (2019) 47–57. · DOI
- Glaucia Melo Edith Law Paulo Alencar and Donald Cowan. 2020. Understanding user understanding: What do developers expect from a cognitive assistant? In Proceedings of the IEEE International Conference on Big Data (Big Data’20) 3165–3172. · DOI
- Ahmad Abdellatif Diego Costa Khaled Badran Rabe Abdalkareem and Emad Shihab. 2020. Challenges in chatbot development: A study of stack overflow posts. In Proceedings of the 17th International Conference on Mining Software Repositories(MSR ’20). Association for Computing Machinery New York NY. 174–185. · DOI
- Ahmad Abdellatif Khaled Badran Diego Elias Costa and Emad Shihab. 2021. A comparison of natural language understanding platforms for chatbots in software engineering. IEEE Trans. Softw. Eng. 48 8 (2021) 3087–3102. · DOI
- Linda Erlenhov Francisco Gomes de Oliveira Neto and Philipp Leitner. 2022. Dependency management bots in open-source systems—Prevalence and adoption. PeerJ Comput. Sci. 8 e849 (2022). · doi:10.7717/peerj-cs.849
- Gabriele De Vito Stefano Lambiase Fabio Palomba and Filomena Ferrucci. 2023. Meet c4se: Your new collaborator for software engineering tasks. In Proceedings of the 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA’23). IEEE Los Alamitos CA. 235–238. · DOI
- Chris Brown and Chris Parnin. 2020. Sorry to bother you again: Developer recommendation choice architectures for designing effective bots. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops(ICSEW’20) Association for Computing Machinery New York NY. 56–60. · DOI
- Anze Gao Yang Zhang Tao Wang Sihao Chen and Jinsheng Deng. 2022. How does bot affect developer’s sentiment: An empirical study on github issues and prs. In Proceedings of the IEEE Smartworld Ubiquitous Intelligence & Computing Scalable Computing & Communications Digital Twin Privacy Computing Metaverse Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta’22). IEEE Los Alamitos CA. 1856–1861. · doi:10.1109/smartworld-uic-atc-scalcom-digitaltwin-pricomp-metaverse56740.2022.00268
- Anze Gao Sihao Chen Tao Wang and Jinsheng Deng. 2022. Understanding the impact of bots on developers sentiment and project progress. In Proceedings of the IEEE 13th International Conference on Software Engineering and Service Science (ICSESS’22). IEEE Los Alamitos CA. 93–96. · DOI
- Sara Pérez-Soler Esther Guerra and Juan de Lara. 2021. Creating and migrating chatbots with conga. In Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion’21). IEEE Los Alamitos CA. 37–40. · DOI
- Jordi Cabot Loli Burgueño Robert Clarisó Gwendal Daniel Jorge Perianez-Pascual and Roberto Rodriguez-Echeverria. 2021. Testing challenges for nlp-intensive bots. In Proceedings of the IEEE/ACM Third International Workshop on Bots in Software Engineering (BotSE’21) 31–34. · DOI
- Charaf Ouaddi Lamya Benaddi and Abdeslam Jakimi. 2024. Architecture tools and dsls for developing conversational agents: An overview. Proc. Comput. Sci. 231 (2024) 293–298. · DOI
- Zhewei Hu and Edward Gehringer. 2023. Bot with interactions: Improving github pull-request feedback through two-way communication. In Proceedings of the IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE’23). IEEE. 28–32. · doi:10.1109/botse59190.2023.00014
- Emily Judith Arteaga Garcia João Felipe Nicolaci Pimentel Zixuan Feng Marco Gerosa Igor Steinmacher and Anita Sarma. 2024. How to support ml end-user programmers through a conversational agent. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12. · DOI
- 10.1111/ijmr.12102 · DOI · Paper page
- Taimour Al Neimat. 2005. Why IT projects fail. Project Perfect White Paper Collect. 8 (2005).
- 10.1016/j.infsof.2023.107375 · DOI
- Frederick P. Brooks Jr. 1995. The Mythical Man-month: Essays on Software Engineering. Pearson Education.
- 10.1016/j.jss.2018.10.002 · DOI
- 10.1145/1610252.1610286 · DOI · Paper page
- 10.1145/1852786.1852788 · DOI · Paper page
- 10.1109/ICSESS58500.2023.10293029 · DOI
- 10.1145/2786805.2786870 · DOI · Paper page
- 10.1002/ev.362 · DOI
- 10.1016/j.infsof.2017.01.001 · DOI · Paper page
- 10.1109/SNPD.2019.8935813 · doi:10.1109/snpd.2019.8935813
- Azham Hussain and Emmanuel O. C. Mkpojiogu. 2016. Requirements: Towards an understanding on why software projects fail. In AIP Conference Proceedings, Vol. 1761. AIP, 020046.
- Project Management Institute. 2021. A Guide to the Project Management Body of Knowledge, 7th ed.
- 10.1145/3305268 · DOI
- 10.1007/s11002-012-9186-1 · DOI · Paper page
- 10.1016/j.infsof.2005.07.002 · DOI · Paper page
- Norri Kageki. 2012. An uncanny mind: Masahiro mori on the uncanny valley and beyond. IEEE Spectrum 12, 06 (2012), 2012.
- Staffs Keele et al. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report. Ver. 2.3 EBSE Technical Report. EBSE.
- 10.1016/j.infsof.2008.09.009 · DOI · Paper page
- 10.1016/j.infsof.2021.106593 · DOI · Paper page
- 10.1109/MS.2017.4541027 · DOI · Paper page
- Tom Lewandowski, Jasmin Delling, Christian Grotherr, and Tilo Böhmann. 2021. State-of-the-art analysis of adopting ai-based conversational agents in organizations: A systematic literature review. In Proceedings of the 25th Pacific Asia Conference on Information Systems. 1–14.
- 10.1145/2635868.2635920 · DOI · Paper page
- Mika V. Mäntylä and Kari Smolander. 2016. Gamification of software testing-an mlr. In International Conference on Product-focused Software Process Improvement. Springer, 611–614.
- 10.1134/S0361768823080145 · DOI
- 10.1145/3527450 · DOI · Paper page
- Dong-Min Park, Seong-Soo Jeong, and Yeong-Seok Seo. 2022. Systematic review on chatbot techniques and applications. J. Inf. Process. Syst. 18, 1 (2022), 26–47.
- 10.3102/00346543061003287 · doi:10.3102/00346543061003287 · Paper page
- 10.1109/MS.2018.290101511 · DOI · Paper page
- 10.1109/ESEM.2009.5316010 · DOI · Paper page
- Amir Shevat. 2017. Designing Bots: Creating Conversational Experiences. O’Reilly Media, Inc.
- 10.1016/j.eswa.2021.115461 · DOI · Paper page
- 10.1109/TCSS.2018.2886433 · DOI
- Richard H. Thaler and Cass R. Sunstein. 2009. Nudge: Improving Decisions About Health, Wealth, and Happiness. Penguin.
- Richard H. Thaler, Cass R. Sunstein, and John P. Balz. 2013. Choice Architecture. Vol. 2013. Princeton University Press Princeton, NJ.
- 10.1145/3476042 · DOI
- 10.1007/978-3-642-29044-2 · DOI · Paper page
Cited by
On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3798166 · Cited by 34
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
Metadata
- Authors
- Lambiase, S., Catolino, G., Palomba, F., & Ferrucci, F.
- Affiliations
- Computer Science, University of Salerno, Fisciano, Italy
- Department of Computer Science, University of Salerno, Fisciano, Italy
- University of Salerno, Fisciano, Italy
- Venue
- ACM Computing Surveys
- Published
- 2024-12-10
- Volume
- 57
- Issue
- 4
- Pages
- 1-37
- ISSN
- 0360-0300
- 1557-7341
- Cited by
- 19
- References
- 9
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review. https://app.cochat.ai/papers/10.1145/3704806
- The paper
- Lambiase, S., Catolino, G., Palomba, F., & Ferrucci, F. (2025). Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review. ACM Computing Surveys. https://doi.org/10.1145/3704806
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.