Automated mobile UI test fragility: an exploratory assessment study on Android
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.
Abstract
Automated UI testing suffers from fragility due to continuous -although minor- changes in the UI of applications. Such fragility has been shown especially for the web domain, though no clear evidence is available for mobile applications. Our goal is to perform an exploratory assessment of the extent and causes of the fragiliy of UI automated tests for mobile applications. For this purpose, we analyzed a small test suite -that we developed using five different testing frameworks for an Android application (K-9 Mail) and observed the changes induced in the tests by the evolution of the UI. We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions. In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons. The preliminary assessment showed that the fragility of UI tests can be a relevant issue also for mobile applications. A few common causes were found that can be used as the basis for providing guidelines for fragility avoidance and repair.
Overview
CoChat Summary
## TL;DR
This exploratory study examined how automated Android UI tests for K-9 Mail break across app versions and found that fragility is a real issue in mobile UI testing too. Across the versions studied, up to 75% of code-based tests and up to 100% of image-recognition tests had to be adapted because of UI evolution.
## Key findings
- The authors wrote 10 test cases for K-9 Mail and implemented them with five frameworks; for the latest version they created 50 distinct test methods in total.
- For code-based tools (Espresso, UIAutomator, Selendroid), the share of tests still working without modification was 5/7 (71%) for v2.995, 8/8 (100%) for v3.309, 6/8 (75%) for v3.993, 2/8 (25%) for v4.804, and 10/10 (100%) for v5.010.
- For Sikuli, the share of unmodified tests still working was 4/7 (57%) for v2.995, 7/8 (87%) for v3.309, 6/8 (75%) for v3.993, 0/8 (0%) for v4.804, and 10/10 (100%) for v5.010.
- The main fragility causes identified were identifier change, text change, deletion or relocation, physical buttons, graphics change, activity flow change, and execution time variability; the paper also notes that Silk Mobile showed weaknesses similar to Espresso/UIAutomator/Selendroid.
## Methods
- Exploratory assessment study on the Android app K-9 Mail, using its localized Italian version and comparing six releases: v2.102, v2.995, v3.309, v3.993, v4.804, and v5.010.
- The test suite covered the app’s main functions: authentication, sending/replying/deleting messages, adding/deleting user accounts, deleting/restoring account data, exporting account settings, and importing account settings.
- The authors implemented and adapted the tests with Espresso, UIAutomator, Selendroid, Sikuli, and also considered Silk Mobile; they then compared which tests remained valid or needed modification across versions.
## Limitations
- The paper explicitly describes the assessment as exploratory and preliminary.
- The study was conducted on a single application (K-9 Mail) and the authors say they used a small test suite.
- They only discuss the tools for which they could write the full set of tests, excluding Silk Mobile from the detailed results because they could not complete the full suite with it.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 13 Not valid · 2 Not checked · 96 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As it can be seen, in this release various actions are available through the physical Menu button of the device. These actions are not distinguished by an identifier.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Authentication , Send a message , Reply to a message , Delete a message test cases are completely compatible with the ones written for release v2.102.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
30 more claims
Create a free account to see all 111 claims and to verify the 3 references still open.
- Probably not a substantial claim. 59
- It may state two separate claims as one. 42
- The quote only partly states this claim. 2
Citations
Cites
Automated Test Input Generation for Android: Are We There Yet? (E)
Choudhary, S. R., Gorla, A., & Orso, A. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.89 · Cited by 463
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346 · cited 1×
“Presented in [21], it was originally intended for the testing of web apps and applications written in Flash.”
MethodAutomating GUI testing for Android applications
Hu, C., & Neamtiu, I. (2011) · Proceedings of the 6th International Workshop on Automation of Software Test · doi:10.1145/1982595.1982612 · Cited by 308 · cited 1×
“Instrumentation tests are particularly fit for automatic UI testing, that is ensuring that the final user does not get improper behaviors when giving input to the app -classifications of typical Android bugs have been given in literature, for instance in [10]- without manually repeating sequences of inputs to the appl…”
Background(2002) · Interlending & Document Supply · doi:10.1108/ilds.2002.12230aab.001 · Cited by 287
MobiGUITAR: Automated Model-Based Testing of Mobile Apps
Amalfitano, D., Fasolino, A. R., Tramontana, P., Ta, B. D., & Memon, A. M. (2015) · IEEE Software · doi:10.1109/ms.2014.55 · Cited by 278
A GUI Crawling-Based Technique for Android Mobile Application Testing
Amalfitano, D., Fasolino, A. R., & Tramontana, P. (2011) · 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2011.77 · Cited by 200 · cited 1×
“sible, possibly running adapted versions of them on a set of different configurations [1].”
Observations and lessons learned from automated testing
Berner, S., Weber, R., & Keller, R. K. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062556 · Cited by 180 · cited 1×
“Fragile tests may trigger extra effort upon failure to: (i) verify no regression has occurred in the system and then (ii) modify the test to adapt it to the changed UI [5].”
BackgroundAutomatically repairing event sequence-based GUI test suites for regression testing
Memon, A. M. (2008) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1416563.1416564 · Cited by 148 · cited 1×
“There are techniques to automate such modification [14] but they are not often applicable or used.”
MethodMobile Application Testing: A Tutorial
Gao, J. Z., Bai, X., Tsai, W., & Uehara, T. (2014) · Computer · doi:10.1109/mc.2013.445 · Cited by 131 · cited 1×
“, voice commands or accelerometer inputs) and devices may have to change their behaviour seamlessly if the battery status or the network to which they are connected change [7].”
BackgroundCapture-replay vs. programmable web testing: An empirical assessment during test case evolution
Leotta, M., Clerissi, D., Ricca, F., & Tonella, P. (2013) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2013.6671302 · Cited by 123 · cited 1×
“Similar evaluations have been done in the field of Web application testing [12, 13].”
Research works printed but not linked: 1
- [16] S. Gunasekaran and V. Bargavi. Survey on automation testing tools for mobile applications. International journal of Advanced Engineering Research and Science (IJIAERS) , 2(11), 2015. -
Cited by
On the Large-scale Graph Data Processing for User Interface Testing in Big Data Science Projects
Uygun, Y., Oguz, R. F., Ölmezoğulları, E., & Aktaş, M. S. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378153 · Cited by 47
“Changes in the UI is a challenge for automated UI testing [14].”
BackgroundMobile GUI Testing Fragility: A Study on Open-Source Android Applications
Coppola, R., Morisio, M., & Torchiano, M. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2869227 · Cited by 47
“In our previous work [8], we developed a small test suite (made of 11 test classes) for K9-Mail3—a popular large-sized open-source Android mail client—and tracked the modifications that were needed by test classes to be executable on different releases.”
MethodSelf-citationA systematic literature review of test breakage prevention and repair techniques
Imtiaz, J., Sherin, S., Khan, M. U., & Iqbal, M. Z. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.05.001 · Cited by 42
A systematic mapping study for graphical user interface testing on mobile apps
Nie, L., Said, K. S., Ma, L., Zheng, Y., & Zhao, Y. (2023) · IET Software · doi:10.1049/sfw2.12123 · Cited by 30
Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes
Coppola, R., Morisio, M., Torchiano, M., & Ardito, L. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09722-9 · Cited by 30
“In our previous works we detailed – as a case study – the maintenance of a small test suite that we developed for a popular Android app (along with a preliminary characterization of fragility causes), finding that modifications performed on its appearance could lead to a need for maintaining up to 75% of the test case…”
MethodSelf-citationGUI testing for mobile applications: objectives, approaches and challenges
Said, K. S., Nie, L., Ajibode, A., & Zhou, X. (2020) · Asia-Pacific Symposium on Internetware · doi:10.1145/3457913.3457931 · Cited by 29
Scripted GUI Testing of Android Apps
Coppola, R., Morisio, M., & Torchiano, M. (2017) · doi:10.1145/3127005.3127008 · Cited by 26
“In our previous study [8] on a popular open-source Android application, K-9 mail, a small test suite was developed and adapted to different releases.”
BackgroundMethodSelf-citationOn the Energy Footprint of Mobile Testing Frameworks
Cruz, L., & Abreu, R. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2946163 · Cited by 29
“Previous work studied five Android testing frameworks in terms of fragilities induced by maintainability [21], [22].”
BackgroundTesting tools for Android context-aware applications: a systematic mapping
de Almeida, D. R., Machado, P. D. L., & Andrade, W. L. (2019) · Journal of the Brazilian Computer Society · doi:10.1186/s13173-019-0093-7 · Cited by 24
Web Canvas Testing Through Visual Inference
Bajammal, M., & Mesbah, A. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00028 · Cited by 20
“More specifically, a visual test consists of a pure image comparison, which does not support the tester in asserting objects on the canvas such as their shape, location, or color, etc; (3) from a maintenance viewpoint, visual testing tools are known to be highly fragile compared to DOM-based methods [4], [5].”
BackgroundMethodMaintenance of Android Widget-Based GUI Testing: A Taxonomy of Test Case Modification Causes
Coppola, R., Morisio, M., & Torchiano, M. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00044 · Cited by 11
“detailed – as a case study – the maintenance of a small test suite that we developed for a popular Android app (along with a preliminary characterization of fragility causes) [2], and gave a quantification of the evolution and fragility of test suites of Android open-source projects hosted on GitHub [3], [4].”
BackgroundSelf-citationTowards automated translation between generations of GUI-based tests for mobile devices
Ardito, L., Coppola, R., Torchiano, M., & Alégroth, E. (2018) · Companion Proceedings for the ISSTA/ECOOP Workshops · doi:10.1145/3236454.3236488 · Cited by 11
“As our sample AUT, similar to a previous case study we conducted to understand the types of fragilities exposed by Scripted Android GUI testing [6], we have chosen K-9 mail, a popular and long-lived (366 releases) open-source e-mail client whose source code is available on GitHub 3 .”
MethodSelf-citationScriptless GUI Testing on Mobile Applications
Jansen, T., Ricós, F. P., Luo, Y., van der Vlist, K., van Dalen, R., Aho, P., & Vos, T. E. J. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00113 · Cited by 12
Repairing Fragile GUI Test Cases Using Word and Layout Embedding
Yoon, J., Chung, S., Shin, K., Kim, J.-H., Hong, S., & Yoo, S. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00038 · Cited by 7
“A GUI test script is fragile because a relatively small change in the appearance of an app can break test cases that traverse the GUI [4].”
BackgroundIntegrating LLM-Based Text Generation with Dynamic Context Retrieval for GUI Testing
Yoon, J., Kim, S., Kim, S., Jung, S. Y., & Yoo, S. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989041 · Cited by 5
“Samsung depends on its own GUI testing framework, and showed that some of the challenges from testing evolving apps developed by 3rd party developers, namely the View Identification Failure (VIF) issues [3], can be effectively addressed with the use of machine learning [4].”
Fragility and evolution of android test suites
Coppola, R. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-c.2017.22 · Cited by 4
“This study has been presented at the INTUITEST workshop, part of ISSTA ‘16 [24].”
MethodSelf-citationResearch on Functional Test of Mobile APP Based on Robot
Pan, Z., Chen, J., Yao, L., & Chen, Z. (2020) · IEEE International Conference on Signal and Image Processing (ICSIP) · doi:10.1109/icsip49896.2020.9339300 · Cited by 4
“Functional automation testing [3]-[7] is mainly to traverse application functions by writing test scripts, and execute them repeatedly.”
MethodAdapTV+: Enhancing Model-Based Test Adaptation for Smart TVs through Icon Recognition
Azimi, M. Y., & Yılmaz, C. (2023) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc59308.2023.00033 · Cited by 3
Rodríguez-Valdés, O., van der Vlist, K., van Dalen, R., Marín, B., & Vos, T. E. J. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-59468-7_10 · Cited by 3
Detection and Segmentation of Graphical Elements on GUIs for Mobile Apps Based on Deep Learning
Hu, R., Chen, M., Cai, L., & Chen, W. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-64214-3_13 · Cited by 4
AdapTV: A Model-Based Test Adaptation Approach for End-to-End User Interface Testing of Smart TVs
Azimi, M. Y., Elgün, C. Ç., Fırat, A., Erata, F., & Yılmaz, C. (2023) · IEEE Access · doi:10.1109/access.2023.3262746 · Cited by 3
“Similarly, the UI testing of mobile applications has also been the subject of numerous studies [45], [46], [47], [48].”
BackgroundModel-based test execution from high-level natural language instructions using GPT-4
Azimi, M. Y., & Yılmaz, C. (2025) · Software Quality Journal · doi:10.1007/s11219-025-09712-9 · Cited by 4
“Amalfi-tano et al. presented a GUI crawling-based technique for Android mobile application testing, which facilitates the automated exploration of app interfaces (Amalfitano et al., 2011).”
On-Device Mobile Application Testing
Vleminckx, I., Parlak, E. B., Kılınççeker, O., & Demeyer, S. (2025) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1109/mobilesoft66462.2025.00013
“Additionally, Android UI Automator and Espresso were used in the test cases to facilitate seamless interaction with the device, as they are among the most popular testing tools, according to Coppola et al. [26].”
A SURVEY OF AUTOMATED TESTING TECHNIQUES FOR ANDROID-BASED MOBILE APPLICATIONS
Eke, N.-O., Salihu, I. A., Usman, A., Ibrahim, R., & Mshelia, Y. U. (2025) · Nigerian Journal of Technology · doi:10.4314/njt.v44i2.15
Slevomat Partner - Android application
Valenta, T. (2020) · Cvut DSpace (Czech Technical University)
Metadata
- Authors
- Coppola, R., Raffero, E., & Torchiano, M.
- Affiliations
- Politecnico di Torino (IT)
- Venue
- Proceedings of the 2nd International Workshop on User Interface Test Automation
- Publisher
- ACM
- Published
- 2016-07-21
- Pages
- 11-20
- OpenAlex
- W2465506749
- Cited by
- 25
- References
- 14
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 3 · 2022: 2 · 2020: 4 · 2019: 7 · 2018: 3 · 2017: 2
- FWCI
- 2.95
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- PORTO@Iris (submittedVersion, other-oa)
- PORTO Publications Open Repository TOrino (Politecnico di Torino) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- mobile application, Android, test maintenance, test repair
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.