CoChat

Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs

Open access
Su, Ting, Yan, Yichen, Wang, Jue, Sun, Jingling, Xiong, Yiheng, Pu, Geguang, Wang, Ke, Su, Zhendong · 2021 · Proceedings of the ACM on Programming Languages
Cited by 60

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Abstract

Android apps are GUI-based event-driven software and have become ubiquitous in recent years. Obviously, functional correctness is critical for an app’s success. However, in addition to crash bugs, non-crashing functional bugs (in short as “non-crashing bugs” in this work) like inadvertent function failures, silent user data lost and incorrect display information are prevalent, even in popular, well-tested apps. These non-crashing functional bugs are usually caused by program logic errors and manifest themselves on the graphic user interfaces (GUIs). In practice, such bugs pose significant challenges in effectively detecting them because (1) current practices heavily rely on expensive, small-scale manual validation ( the lack of automation ); and (2) modern fully automated testing has been limited to crash bugs ( the lack of test oracles ). This paper fills this gap by introducing independent view fuzzing , a novel, fully automated approach for detecting non-crashing functional bugs in Android apps. Inspired by metamorphic testing, our key insight is to leverage the commonly-held independent view property of Android apps to manufacture property-preserving mutant tests from a set of seed tests that validate certain app properties. The mutated tests help exercise the tested apps under additional, adverse conditions. Any property violations indicate likely functional bugs for further manual confirmation. We have realized our approach as an automated, end-to-end functional fuzzing tool, Genie. Given an app, (1) Genie automatically detects non-crashing bugs without requiring human-provided tests and oracles (thus fully automated ); and (2) the detected non-crashing bugs are diverse (thus general and not limited to specific functional properties ), which set Genie apart from prior work. We have evaluated Genie on 12 real-world Android apps and successfully uncovered 34 previously unknown non-crashing bugs in their latest releases — all have been confirmed, and 22 have already been fixed. Most of the detected bugs are nontrivial and have escaped developer (and user) testing for at least one year and affected many app releases, thus clearly demonstrating Genie’s effectiveness. According to our analysis, Genie achieves a reasonable true positive rate of 40.9%, while these 34 non-crashing bugs could not be detected by prior fully automated GUI testing tools (as our evaluation confirms). Thus, our work complements and enhances existing manual testing and fully automated testing for crash bugs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3485533

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • The oracle problem in software testing: A survey

    Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104

  • A survey on tree edit distance and related problems

    Bille, P. (2005) · Theoretical Computer Science · doi:10.1016/j.tcs.2004.12.030 · Cited by 751 · cited 1×

    “In practice, to efficiently compute Δ ( ℓ 𝑖 𝑗 ) , we adapt the classic tree edit distance algorithms [Bille 2005; Pawlik and Augsten 2016; Zhang and Shasha 1989] to compute the minimal edit operations between the two ordered GUI trees 𝑇 𝑖 and 𝑇 𝑗 (corresponding to ℓ 𝑖 and ℓ 𝑗 ).”
  • Compiler validation via equivalence modulo inputs

    Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388 · cited 1×

    “It has been widely used in testing many other specification-less software systems, e.g. , compilers [Le et al. 2014; Sun et al. 2016], debuggers [Tolksdorf et al. 2019], and machine learning-based systems [Tian et al. 2018; Xie et al. 2011].”
    Self-citation

Cited by

See all 60

Metadata

Authors
Su, T., Yan, Y., Wang, J., Sun, J., Xiong, Y., Pu, G., Wang, K., & Su, Z.
Venue
Proceedings of the ACM on Programming Languages
Published
2021-10-15
Volume
5
Issue
OOPSLA
Pages
1-31
ISSN
2475-1421 (ISSN-L)
OpenAlex
W3206846523
Cited by
60
References
3
FWCI
13.96
Type
journal article
Language
en
Open access
diamond

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.