CoChat

Automated mobile UI test fragility: an exploratory assessment study on Android

Open access
Coppola, Riccardo, Raffero, Emanuele, Torchiano, Marco · 2016 · Proceedings of the 2nd International Workshop on User Interface Test Automation

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.

Abstract

Automated UI testing suffers from fragility due to continuous -although minor- changes in the UI of applications. Such fragility has been shown especially for the web domain, though no clear evidence is available for mobile applications. Our goal is to perform an exploratory assessment of the extent and causes of the fragiliy of UI automated tests for mobile applications. For this purpose, we analyzed a small test suite -that we developed using five different testing frameworks for an Android application (K-9 Mail) and observed the changes induced in the tests by the evolution of the UI. We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions. In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons. The preliminary assessment showed that the fragility of UI tests can be a relevant issue also for mobile applications. A few common causes were found that can be used as the basis for providing guidelines for fragility avoidance and repair.

Overview

CoChat Summary

## TL;DR

This exploratory study examined how automated Android UI tests for K-9 Mail break across app versions and found that fragility is a real issue in mobile UI testing too. Across the versions studied, up to 75% of code-based tests and up to 100% of image-recognition tests had to be adapted because of UI evolution.

## Key findings

  • The authors wrote 10 test cases for K-9 Mail and implemented them with five frameworks; for the latest version they created 50 distinct test methods in total.
  • For code-based tools (Espresso, UIAutomator, Selendroid), the share of tests still working without modification was 5/7 (71%) for v2.995, 8/8 (100%) for v3.309, 6/8 (75%) for v3.993, 2/8 (25%) for v4.804, and 10/10 (100%) for v5.010.
  • For Sikuli, the share of unmodified tests still working was 4/7 (57%) for v2.995, 7/8 (87%) for v3.309, 6/8 (75%) for v3.993, 0/8 (0%) for v4.804, and 10/10 (100%) for v5.010.
  • The main fragility causes identified were identifier change, text change, deletion or relocation, physical buttons, graphics change, activity flow change, and execution time variability; the paper also notes that Silk Mobile showed weaknesses similar to Espresso/UIAutomator/Selendroid.

## Methods

  • Exploratory assessment study on the Android app K-9 Mail, using its localized Italian version and comparing six releases: v2.102, v2.995, v3.309, v3.993, v4.804, and v5.010.
  • The test suite covered the app’s main functions: authentication, sending/replying/deleting messages, adding/deleting user accounts, deleting/restoring account data, exporting account settings, and importing account settings.
  • The authors implemented and adapted the tests with Espresso, UIAutomator, Selendroid, Sikuli, and also considered Silk Mobile; they then compared which tests remained valid or needed modification across versions.

## Limitations

  • The paper explicitly describes the assessment as exploratory and preliminary.
  • The study was conducted on a single application (K-9 Mail) and the authors say they used a small test suite.
  • They only discuss the tools for which they could write the full set of tests, excluding Silk Mobile from the detailed results because they could not complete the full suite with it.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/2945404.2945406

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 13 Not valid · 2 Not checked · 96 Own results and interpretations

30 more claims

Create a free account to see all 111 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 59
  • It may state two separate claims as one. 42
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Citations

Cites

Show all 14 references

Research works printed but not linked: 1

  1. [16] S. Gunasekaran and V. Bargavi. Survey on automation testing tools for mobile applications. International journal of Advanced Engineering Research and Science (IJIAERS) , 2(11), 2015. -

Cited by

See all 25

Metadata

Authors
Coppola, R., Raffero, E., & Torchiano, M.
Affiliations
Politecnico di Torino (IT)
Venue
Proceedings of the 2nd International Workshop on User Interface Test Automation
Publisher
ACM
Published
2016-07-21
Pages
11-20
OpenAlex
W2465506749
Cited by
25
References
14
Citations by year
2025: 3 · 2024: 1 · 2023: 3 · 2022: 2 · 2020: 4 · 2019: 7 · 2018: 3 · 2017: 2
FWCI
2.95
Type
conference paper
Language
en
Open access
Locations
  • PORTO@Iris (submittedVersion, other-oa)
  • PORTO Publications Open Repository TOrino (Politecnico di Torino) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
mobile application, Android, test maintenance, test repair

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automated mobile UI test fragility: an exploratory assessment study on Android. https://app.cochat.ai/papers/10.1145/2945404.2945406
The paper
Coppola, R., Raffero, E., & Torchiano, M. (2016). Automated mobile UI test fragility: an exploratory assessment study on Android. Proceedings of the 2nd International Workshop on User Interface Test Automation. https://doi.org/10.1145/2945404.2945406

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.