CoChat

Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions

Open access
Liu, Zhe, Chen, Chunyang, Wang, Junjie, Chen, Mengzhuo, Wu, Boyu, Che, Xing, Wang, Dandan, Wang, Qing · 2024 · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Cited by 125

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

Automated Graphical User Interface (GUI) testing plays a crucial role in ensuring app quality, especially as mobile applications have become an integral part of our daily lives. Despite the growing popularity of learning-based techniques in automated GUI testing due to their ability to generate human-like interactions, they still suffer from several limitations, such as low testing coverage, inadequate generalization capabilities, and heavy reliance on training data. Inspired by the success of Large Language Models (LLMs) like ChatGPT in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within this framework, we have also introduced a functionality-aware memory prompting mechanism that equips the LLM with the ability to retain testing knowledge of the whole process and conduct long-term, functionality-based reasoning to guide exploration. We evaluate it on 93 apps from Google Play and demonstrate that it outperforms the best baseline by 32% in activity coverage, and detects 31% more bugs at a faster rate. Moreover, GPTDroid identifies 53 new bugs on Google Play, of which 35 have been confirmed and fixed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3597503.3639180

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 66

  1. Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2023. A3Test: Assertion-Augmented Automated Test Case Generation. arXiv preprint arXiv:2302.10352 (2023). · arXiv
  2. 10.1002/spe.2564 · doi:10.1002/spe.2564
  3. Farnaz Behrang and Alessandro Orso. 2019. Test Migration Between Mobile Apps with Similar Functionality. In ASE 2019.
  4. 10.1145/3387903.3389308 · DOI
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
  7. Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2022. Fuzzing Deep-Learning Libraries via Large Language Models. arXiv preprint arXiv:2212.14834 (2022). · arXiv
  8. Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of android apps. In ICSE. IEEE.
  9. Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2018. Large-scale analysis of framework-specific exceptions in android apps. In ICSE. IEEE, 408--419.
  10. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. EMNLP (2020).
  11. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022). · arXiv
  12. Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In ICSE. IEEE.
  13. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Ppt: Pre-trained prompt tuning for few-shot learning. (2021).
  14. 10.1109/SP40000.2020.00071 · DOI
  15. Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2023. Explainable Automated Debugging via Large Language Model-driven Scientific Debugging. arXiv preprint arXiv:2304.02195 (2023). · arXiv
  16. Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2022. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. CoRR abs/2209.11515 (2022). · arXiv
  17. 10.1109/TR.2018.2865733 · DOI · Paper page
  18. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping coverage plateaus in test generation with pretrained large language models. (2023).
  19. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In ICSE. IEEE.
  20. 10.1109/ASE.2019.00104 · DOI
  21. 10.1145/3477495.3532048 · DOI
  22. Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Marques, and Lingfei Zeng. 2017. Automatic text input generation for mobile testing. In ICSE. IEEE.
  23. Zhe Liu Chunyang Chen Junjie Wang Xing Che Yuekai Huang Jun Hu and Qing Wang. 2023. Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing. In ICSE. · DOI · Paper page
  24. 10.1109/TSE.2022.3150876 · DOI · Paper page
  25. 10.1109/ICSE48619.2023.00168 · DOI
  26. 10.1145/3510454.3516848 · DOI
  27. 10.1145/3491102.3501903 · DOI
  28. Zhengwei Lv Chao Peng Zhao Zhang Ting Su Kai Liu and Ping Yang. 2022. Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning. In ICSE.
  29. 10.1145/2491411.2491450 · DOI · Paper page
  30. 10.1145/2931037.2931054 · DOI · Paper page
  31. 10.1109/TSE.2022.3183297 · DOI · Paper page
  32. 10.1109/ICSE43902.2021.00041 · DOI · Paper page
  33. Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. In ICSE. IEEE.
  34. Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J Mooney, and Milos Gligoric. 2023. Learning Deep Semantics for Test Completion. arXiv preprint arXiv:2302.10166 (2023). · arXiv
  35. 10.1145/3395363.3397354 · DOI · Paper page
  36. Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2022. Examining Zero-Shot Vulnerability Repair with Large Language Models. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 1--18.
  37. 10.1007/s10664-021-10059-5 · DOI
  38. 10.1109/ICSME55016.2022.00074 · DOI
  39. Andrea Romdhana, Alessio Merlo, Mariano Ceccato, and Paolo Tonella. 2022. Deep reinforcement learning for black-box testing of android apps. TOSEM (2022).
  40. 10.1109/ASE.2015.66 · doi:10.1109/ase.2015.66
  41. 10.1109/MC.2018.2141024 · doi:10.1109/mc.2018.2141024
  42. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. Adaptive test generation using a large language model. arXiv preprint arXiv:2302.06527 (2023). · arXiv
  43. Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. 2023. Exploring the Effectiveness of Large Language Models in Generating Unit Tests. arXiv preprint arXiv:2305.00418 (2023). · arXiv
  44. 10.1145/3106237.3106298 · DOI · Paper page
  45. Saghar Talebipour, Yixue Zhao, Luka Dojcilovic, Chenggang Li, and Nenad Medvidovic. 2021. UI Test Migration Across Mobile Platforms. In ASE. IEEE, 756--767.
  46. 10.1145/3377811.3380349 · DOI
  47. 10.1145/3524481.3527220 · DOI · Paper page
  48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (2017).
  49. 10.1145/3472749.3474765 · DOI · Paper page
  50. Jue Wang Yanyan Jiang Chang Xu Chun Cao Xiaoxing Ma and Jian Lu. 2020. Combodroid: generating high-quality test inputs for android apps via use case combinations. In ICSE. 469--480. · DOI
  51. 10.1145/3460319.3464828 · doi:10.1145/3460319.3464828
  52. Wenyu Wang Wei Yang Tianyin Xu and Tao Xie. 2021. Vet: identifying and avoiding UI exploration tarpits. In FSE. 83--94. · DOI
  53. Qing Xie and Atif M Memon. 2007. Designing and comparing automated test oracles for GUI-based software applications. TOSEM (2007).
  54. Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023. ChatUniTest: a ChatGPT-based automated unit test generation tool. arXiv preprint arXiv:2305.04764 (2023). · arXiv
  55. 10.5555/3288647.3288710 · doi:10.5555/3288647.3288710
  56. 10.1007/978-3-642-37057-1_19 · DOI · Paper page
  57. Husam N Yasin, Siti Hafizah Ab Hamid, and Raja Jamilah Raja Yusof. 2021. Droidbotx: Test case generation tool for android applications using Q-learning. Symmetry (2021).
  58. 10.1145/3533767.3534219 · DOI · Paper page
  59. Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation. arXiv preprint arXiv:2305.04207 (2023). · arXiv
  60. Xia Zeng Dengfeng Li Wujie Zheng Fan Xia Yuetang Deng Wing Lam Wei Yang and Tao Xie. 2016. Automated test input generation for android: Are we really there yet in an industrial case?. In FSE. · doi:10.1145/2950290.2983958
  61. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022). · arXiv
  62. 10.1145/3540250.3558934 · DOI
  63. 10.1145/3411764.3445186 · DOI · Paper page
  64. Liu Zhe, Chen Chunyang, Wang Junjie, Huang Yuekai, Hu Jun, and Wang Qing. 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In 2020 35rd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE.
  65. Haibing Zheng, Dengfeng Li, Beihai Liang, Xia Zeng, Wujie Zheng, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. 2017. Automated test input generation for android: Towards getting there in an industrial case. In ICSE. IEEE.
  66. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision (2022), 1--12.

Cited by

See all 125

Metadata

Authors
Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Che, X., Wang, D., & Wang, Q.
Venue
Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Publisher
ACM
Published
2024-04-12
Pages
1-13
OpenAlex
W4394769323
Cited by
125
References
8
FWCI
35.35
Type
conference paper
Language
en
Open access
hybrid
Funders
National Natural Science Foundation of China (No.62232016, No.62072442, No.62272445)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions. https://app.cochat.ai/papers/10.1145/3597503.3639180
The paper
Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Che, X., Wang, D., & Wang, Q. (2024). Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. https://doi.org/10.1145/3597503.3639180

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.