The DOI matches its Crossref record.
Understanding the Test Automation Culture of App Developers
Open accessAbstract
Smartphone applications (apps) have gained popularity recently. Millions of smartphone applications (apps) are available on different app stores which gives users plethora of options to choose from, however, it also raises concern if these apps are adequately tested before they are released for public use. In this study, we want to understand the test automation culture prevalent among app developers. Specifically, we want to examine the current state of testing of apps, the tools that are commonly used by app developers, and the problems faced by them. To get an insight on the test automation culture, we conduct two different studies. In the first study, we analyse over 600 Android apps collected from F- Droid, one of the largest repositories containing information about open-source Android apps. We check for the presence of test cases and calculate code coverage to measure the adequacy of testing in these apps. We also survey developers who have hosted their applications on GitHub to understand the testing practices followed by them. We ask developers about the tools that they use and ''pain points'' that they face while testing Android apps. For the second study, based on the responses from Android developers, we improve our survey questions and resend it to Windows app developers within Microsoft. We conclude that many Android apps are poorly tested - only about 14% of the apps contain test cases and only about 9% of the apps that have executable test cases have coverage above 40\%. Also, we find that Android app developers use automated testing tools such as JUnit, Monkeyrunner, Robotium, and Robolectric, however, they often prefer to test their apps manually, whereas Windows app developers prefer to use in-house tools such as Visual Studio and Microsoft Test Manager. Both Android and Windows app developers face many challenges such as time constraints, compatibility issues, lack of exposure, cumbersome tools, etc. We give suggestions to improve the test automation culture in the growing app community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Sapienz: multi-objective automated testing for Android applications
Mao, K., Harman, M., & Jia, Y. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931054 · Cited by 541
AXNav: Replaying Accessibility Tests from Natural Language
Taeb, M., Swearngin, A., Schoop, E., Cheng, R., Jiang, Y., & Nichols, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642777 · Cited by 52
“However, manual testing comes at a huge cost and thus QA and development have developed test suites for automating UI tests [33].”
BackgroundRazzaq, A., Buckley, J., Lai, Q., Yu, T., & Botterweck, G. (2025) · ACM Computing Surveys · doi:10.1145/3687299 · Cited by 36
Exploration of technical debt in start-ups
Klotins, E., Unterkalmsteiner, M., Chatzipetrou, P., Gorschek, T., Prikladnicki, R., Tripathi, N., & Pompermaier, L. B. (2018) · Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3183519.3183539 · Cited by 24
Sun, Z., Du, X., Song, F., Wang, S., Ni, M., Li, L., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688831 · Cited by 8
Self-citation
Metadata
- Authors
- Kochhar, P. S., Thung, F., Nagappan, N., Zimmermann, T., & Lo, D.
- Venue
- 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)
- Publisher
- IEEE
- Published
- 2015-04
- Pages
- 1-10
- OpenAlex
- W2146511370
- Cited by
- 159
- References
- 0
- FWCI
- 14.22
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding the Test Automation Culture of App Developers. https://app.cochat.ai/papers/10.1109/icst.2015.7102609
- The paper
- Kochhar, P. S., Thung, F., Nagappan, N., Zimmermann, T., & Lo, D. (2015). Understanding the Test Automation Culture of App Developers. 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). https://doi.org/10.1109/icst.2015.7102609
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.