CoChat

Testing the Limits: Unusual Text Inputs Generation for Mobile App Crash Detection with Large Language Model

Liu, Zhe, Chen, Chunyang, Wang, Junjie, Chen, Mengzhuo, Wu, Boyu, Tian, Zhilin, Huang, Yuekai, Hu, Jun, Wang, Qing · 2024 · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Cited by 58

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Abstract

Mobile applications have become a ubiquitous part of our daily life, providing users with access to various services and utilities. Text input, as an important interaction channel between users and applications, plays an important role in core functionality such as search queries, authentication, messaging, etc. However, certain special text (e.g., -18 for Font Size) can cause the app to crash, and generating diversified unusual inputs for fully testing the app is highly demanded. Nevertheless, this is also challenging due to the combination of explosion dilemma, high context sensitivity, and complex constraint relations. This paper proposes InputBlaster which leverages the LLM to automatically generate unusual text inputs for mobile app crash detection. It formulates the unusual inputs generation problem as a task of producing a set of test generators, each of which can yield a batch of unusual text inputs under the same mutation rule. In detail, InputBlaster leverages LLM to produce the test generators together with the mutation rules serving as the reasoning chain, and utilizes the in-context learning schema to demonstrate the LLM with examples for boosting the performance. InputBlaster is evaluated on 36 text input widgets with cash bugs involving 31 popular Android apps, and results show that it achieves 78% bug detection rate, with 136% higher than the best baseline. Besides, we integrate it with the automated GUI testing tool and detect 37 unseen crashes in real-world apps.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3597503.3639118

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 57

  1. Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training LLMs for project-specific code-summarization. ASE (2022).
  2. Nadia Alshahwan and Mark Harman. 2011. Automated web application testing using search based software engineering. In ASE. IEEE, 3--12.
  3. 10.1145/2393596.2393666 · DOI · Paper page
  4. 10.1109/APSEC.2016.036 · doi:10.1109/apsec.2016.036
  5. 10.1002/spe.2564 · doi:10.1002/spe.2564
  6. 10.1145/2509136.2509549 · DOI
  7. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901.
  8. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901.
  9. 10.1145/3387903.3389308 · DOI
  10. 10.1016/j.cose.2018.02.002 · DOI
  11. 10.1145/3498707 · DOI
  12. 10.1007/978-3-030-59152-6_18 · doi:10.1007/978-3-030-59152-6_18
  13. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. 2020. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems 33 (2020), 22243--22255.
  14. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
  15. 10.1007/978-3-030-30806-3_8 · doi:10.1007/978-3-030-30806-3_8
  16. Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT. ISSTA (2023).
  17. 10.1145/3377811.3380402 · DOI
  18. 10.1109/ISSRE.2013.6698888 · DOI
  19. 10.1109/ICSE.2019.00042 · doi:10.1109/icse.2019.00042
  20. 10.1109/SP40000.2020.00071 · DOI
  21. Lukav Holik, Petr Jank, Anthony W Lin, Philipp Rmmer, and Tom Vojnar. 2017. String constraints with concatenation and transducers solved efficiently. Proceedings of the ACM on Programming Languages 2, POPL (2017), 1--32.
  22. 10.1109/ASE.2019.00044 · DOI · Paper page
  23. Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. ICSE (2023).
  24. 10.1145/2377656.2377662 · DOI
  25. Sebastian Krings, Joshua Schmidt, Patrick Skowronek, Jannik Dunkelau, and Dierk Ehmke. 2020. Towards constraint logic programming over strings for test data generation. In Declarative Programming and Knowledge Management: Conference on Declarative Programming, DECLARE 2019, Unifying INAP, WLP, and WFLP, Cottbus, Germany, September 9--12, 2019, Revised Selected Papers 22. Springer, 139--159.
  26. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pretrained Large Language Models. In ICSE.
  27. 10.1007/978-3-319-03077-7_2 · DOI
  28. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. DroidBot: A Lightweight UI-Guided Test Input Generator for Android (ICSE-C '17).
  29. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 23--26.
  30. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: a deep learning-based approach to automated black-box Android app testing. In ASE. IEEE, 1070--1073.
  31. 10.1109/TR.2018.2834476 · DOI · Paper page
  32. 10.1007/978-3-319-08867-9_43 · doi:10.1007/978-3-319-08867-9_43
  33. 10.1109/ICSE.2017.65 · DOI · Paper page
  34. Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing. arXiv preprint arXiv:2212.04732 (2022). · arXiv
  35. 10.1145/3324884.3416547 · DOI · Paper page
  36. 10.1109/TSE.2022.3150876 · DOI · Paper page
  37. 10.1109/ICSE48619.2023.00168 · DOI
  38. 10.1145/3510454.3516848 · DOI
  39. 10.1145/3491102.3501903 · DOI
  40. 10.1145/2491411.2491450 · DOI · Paper page
  41. 10.1145/2931037.2931054 · DOI · Paper page
  42. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. Computer Science (2013).
  43. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837 (2022). · arXiv
  44. 10.1145/3395363.3397354 · DOI · Paper page
  45. Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. ICLR (2022).
  46. 10.1109/ICSTW.2017.21 · doi:10.1109/icstw.2017.21
  47. 10.1145/2435349.2435379 · DOI · Paper page
  48. 10.1109/32.799955 · DOI · Paper page
  49. 10.1007/s40593-022-00300-7 · DOI · Paper page
  50. 10.1145/3106237.3106298 · DOI · Paper page
  51. Nezih Sunman, Yiğit Soydan, and Hasan Sözer. 2022. Automated web application testing driven by pre-recorded test cases. Journal of Systems and Software (2022), 111441.
  52. 10.1145/2660267.2660372 · DOI
  53. 10.1007/978-3-319-63390-9_21 · DOI
  54. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (2017).
  55. Jue Wang Yanyan Jiang Chang Xu Chun Cao Xiaoxing Ma and Jian Lu. 2020. Combodroid: generating high-quality test inputs for android apps via use case combinations. In ICSE. 469--480. · DOI
  56. 10.1609/aaai.v36i3.20215 · DOI · Paper page
  57. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022). · arXiv

Cited by

See all 58

Metadata

Authors
Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Tian, Z., Huang, Y., Hu, J., & Wang, Q.
Affiliations
  • University of Chinese Academy of Sciences, Beijing, China
  • Technical University of Munich, Munich, Germany
  • Institute of Software, Chinese Academy of Sciences, Beijing, China
  • Pennsylvania State University, State College, Pennsylvania, USA
Venue
Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Publisher
ACM
Published
2024-04-12
Pages
1-12
Cited by
58
References
9
Funders
National Natural Science Foundation of China (No.62232016, No.62072442, No.62272445)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.