Automated mobile UI test fragility: an exploratory assessment study on Android
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.
Abstract
Automated UI testing suffers from fragility due to continuous -although minor- changes in the UI of applications. Such fragility has been shown especially for the web domain, though no clear evidence is available for mobile applications. Our goal is to perform an exploratory assessment of the extent and causes of the fragiliy of UI automated tests for mobile applications. For this purpose, we analyzed a small test suite -that we developed using five different testing frameworks for an Android application (K-9 Mail) and observed the changes induced in the tests by the evolution of the UI. We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions. In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons. The preliminary assessment showed that the fragility of UI tests can be a relevant issue also for mobile applications. A few common causes were found that can be used as the basis for providing guidelines for fragility avoidance and repair.
Overview
CoChat Summary
## TL;DR
This exploratory study examined how automated Android UI tests for K-9 Mail break across app versions and found that fragility is a real issue in mobile UI testing too. Across the versions studied, up to 75% of code-based tests and up to 100% of image-recognition tests had to be adapted because of UI evolution.
## Key findings
- The authors wrote 10 test cases for K-9 Mail and implemented them with five frameworks; for the latest version they created 50 distinct test methods in total.
- For code-based tools (Espresso, UIAutomator, Selendroid), the share of tests still working without modification was 5/7 (71%) for v2.995, 8/8 (100%) for v3.309, 6/8 (75%) for v3.993, 2/8 (25%) for v4.804, and 10/10 (100%) for v5.010.
- For Sikuli, the share of unmodified tests still working was 4/7 (57%) for v2.995, 7/8 (87%) for v3.309, 6/8 (75%) for v3.993, 0/8 (0%) for v4.804, and 10/10 (100%) for v5.010.
- The main fragility causes identified were identifier change, text change, deletion or relocation, physical buttons, graphics change, activity flow change, and execution time variability; the paper also notes that Silk Mobile showed weaknesses similar to Espresso/UIAutomator/Selendroid.
## Methods
- Exploratory assessment study on the Android app K-9 Mail, using its localized Italian version and comparing six releases: v2.102, v2.995, v3.309, v3.993, v4.804, and v5.010.
- The test suite covered the app’s main functions: authentication, sending/replying/deleting messages, adding/deleting user accounts, deleting/restoring account data, exporting account settings, and importing account settings.
- The authors implemented and adapted the tests with Espresso, UIAutomator, Selendroid, Sikuli, and also considered Silk Mobile; they then compared which tests remained valid or needed modification across versions.
## Limitations
- The paper explicitly describes the assessment as exploratory and preliminary.
- The study was conducted on a single application (K-9 Mail) and the authors say they used a small test suite.
- They only discuss the tools for which they could write the full set of tests, excluding Silk Mobile from the detailed results because they could not complete the full suite with it.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 13 Not valid · 2 Not checked · 96 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As it can be seen, in this release various actions are available through the physical Menu button of the device. These actions are not distinguished by an identifier.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Authentication , Send a message , Reply to a message , Delete a message test cases are completely compatible with the ones written for release v2.102.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
30 more claims
Create a free account to see all 111 claims and to verify the 3 references still open.
- Probably not a substantial claim. 59
- It may state two separate claims as one. 42
- The quote only partly states this claim. 2
Citations
Cites
Automated Test Input Generation for Android: Are We There Yet? (E)
Choudhary, S. R., Gorla, A., & Orso, A. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.89 · Cited by 463
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346 · cited 1×
“Presented in [21], it was originally intended for the testing of web apps and applications written in Flash.”
MethodAutomating GUI testing for Android applications
Hu, C., & Neamtiu, I. (2011) · Proceedings of the 6th International Workshop on Automation of Software Test · doi:10.1145/1982595.1982612 · Cited by 308 · cited 1×
“Instrumentation tests are particularly fit for automatic UI testing, that is ensuring that the final user does not get improper behaviors when giving input to the app -classifications of typical Android bugs have been given in literature, for instance in [10]- without manually repeating sequences of inputs to the appl…”
Background(2002) · Interlending & Document Supply · doi:10.1108/ilds.2002.12230aab.001 · Cited by 287
MobiGUITAR: Automated Model-Based Testing of Mobile Apps
Amalfitano, D., Fasolino, A. R., Tramontana, P., Ta, B. D., & Memon, A. M. (2015) · IEEE Software · doi:10.1109/ms.2014.55 · Cited by 278
A GUI Crawling-Based Technique for Android Mobile Application Testing
Amalfitano, D., Fasolino, A. R., & Tramontana, P. (2011) · 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2011.77 · Cited by 200 · cited 1×
“sible, possibly running adapted versions of them on a set of different configurations [1].”
Observations and lessons learned from automated testing
Berner, S., Weber, R., & Keller, R. K. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062556 · Cited by 180 · cited 1×
“Fragile tests may trigger extra effort upon failure to: (i) verify no regression has occurred in the system and then (ii) modify the test to adapt it to the changed UI [5].”
BackgroundAutomatically repairing event sequence-based GUI test suites for regression testing
Memon, A. M. (2008) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1416563.1416564 · Cited by 148 · cited 1×
“There are techniques to automate such modification [14] but they are not often applicable or used.”
MethodMobile Application Testing: A Tutorial
Gao, J. Z., Bai, X., Tsai, W., & Uehara, T. (2014) · Computer · doi:10.1109/mc.2013.445 · Cited by 131 · cited 1×
“, voice commands or accelerometer inputs) and devices may have to change their behaviour seamlessly if the battery status or the network to which they are connected change [7].”
BackgroundCapture-replay vs. programmable web testing: An empirical assessment during test case evolution
Leotta, M., Clerissi, D., Ricca, F., & Tonella, P. (2013) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2013.6671302 · Cited by 123 · cited 1×
“Similar evaluations have been done in the field of Web application testing [12, 13].”
Menko, F. H., Vasen, H. F. A., & Lynch, H. T. (2001) · Annals of Oncology · doi:10.1093/oxfordjournals.annonc.a000270 · Cited by 90
Visual vs. DOM-Based Web Locators: An Empirical Study
Leotta, M., Clerissi, D., Ricca, F., & Tonella, P. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-08245-5_19 · Cited by 74 · cited 1×
“Similar evaluations have been done in the field of Web application testing [12, 13].”
Android application development and testability
Knych, T. W., & Baliga, A. (2014) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1145/2593902.2593910 · Cited by 15 · cited 1×
“The framework uses synchronization mechanisms, and is capable to check whether activities are actually in a stable state before performing operations on views [11].”
BackgroundAutomated testing tools for mobile applications
Saad, N. H., & Bakar, N. S. A. A. (2014) · International Conference on Information and Communication Technology for the Muslim World (ICT4M) · doi:10.1109/ict4m.2014.7020665 · Cited by 11
Research works printed but not linked: 1
- [16] S. Gunasekaran and V. Bargavi. Survey on automation testing tools for mobile applications. International journal of Advanced Engineering Research and Science (IJIAERS) , 2(11), 2015. -
Cited by
On the Large-scale Graph Data Processing for User Interface Testing in Big Data Science Projects
Uygun, Y., Oguz, R. F., Ölmezoğulları, E., & Aktaş, M. S. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378153 · Cited by 47
“Changes in the UI is a challenge for automated UI testing [14].”
BackgroundMobile GUI Testing Fragility: A Study on Open-Source Android Applications
Coppola, R., Morisio, M., & Torchiano, M. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2869227 · Cited by 47
“In our previous work [8], we developed a small test suite (made of 11 test classes) for K9-Mail3—a popular large-sized open-source Android mail client—and tracked the modifications that were needed by test classes to be executable on different releases.”
MethodSelf-citationA systematic literature review of test breakage prevention and repair techniques
Imtiaz, J., Sherin, S., Khan, M. U., & Iqbal, M. Z. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.05.001 · Cited by 42
A systematic mapping study for graphical user interface testing on mobile apps
Nie, L., Said, K. S., Ma, L., Zheng, Y., & Zhao, Y. (2023) · IET Software · doi:10.1049/sfw2.12123 · Cited by 30
Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes
Coppola, R., Morisio, M., Torchiano, M., & Ardito, L. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09722-9 · Cited by 30
“In our previous works we detailed – as a case study – the maintenance of a small test suite that we developed for a popular Android app (along with a preliminary characterization of fragility causes), finding that modifications performed on its appearance could lead to a need for maintaining up to 75% of the test case…”
MethodSelf-citation
Metadata
- Authors
- Coppola, R., Raffero, E., & Torchiano, M.
- Affiliations
- Politecnico di Torino (IT)
- Venue
- Proceedings of the 2nd International Workshop on User Interface Test Automation
- Publisher
- ACM
- Published
- 2016-07-21
- Pages
- 11-20
- OpenAlex
- W2465506749
- Cited by
- 25
- References
- 14
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 3 · 2022: 2 · 2020: 4 · 2019: 7 · 2018: 3 · 2017: 2
- FWCI
- 2.95
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- PORTO@Iris (submittedVersion, other-oa)
- PORTO Publications Open Repository TOrino (Politecnico di Torino) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- mobile application, Android, test maintenance, test repair
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.