CoChat
Sign Up
PreprintCited by 141Advanced Malware Detection TechniquesInteractive and Immersive DisplaysMobile and Web Applications

The DOI matches its DataCite record.

Android in the Wild: A Large-Scale Dataset for Android Device Control

Open access
Rawles, Christopher, Li, Alice, Rodriguez, Daniel, Riva, Oriana, Lillicrap, Timothy·2023·arXiv
DOI 10.48550/arxiv.2307.10088arXiv 2307.10088
Open this paper in CoChat — free accountOpen PDF

Abstract

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.10088

Cites

Statistics

Figures

Figure 2 : Figure 2: Statistics for AITW. a) Episode length distribution. b) Episode length distribution by dataset group. c) Frequency of Android apps in the dataset. d) Token analysis including distribution of instruction length and token frequency for GOOGLEAPPS and GENERAL.
Figure 2 : Figure 2: Statistics for AITW. a) Episode length distribution. b) Episode length distribution by dataset group. c) Frequency of Android apps in the dataset. d) Token analysis including distribution of instruction length and token frequency for GOOGLEAPPS and GENERAL.CC-BY · From the PDF
Figure 3 : Figure 3: True complete match (estimated using human evaluation), and partial and complete match (both estimated using automated evaluation) for BC-history. True complete match is based on a subset of episodes; 95% confidence bounds are reported. Partial match is correlated with true complete match, while the complete match heuristic is a lower bound score.
Figure 3 : Figure 3: True complete match (estimated using human evaluation), and partial and complete match (both estimated using automated evaluation) for BC-history. True complete match is based on a subset of episodes; 95% confidence bounds are reported. Partial match is correlated with true complete match, while the complete match heuristic is a lower bound score.CC-BY · From the PDF
Figure 4 : Figure 4: Example episode from the dataset.
Figure 4 : Figure 4: Example episode from the dataset.CC-BY · From the PDF
Figure 5 : Figure 5: Example episode from the dataset.
Figure 5 : Figure 5: Example episode from the dataset.CC-BY · From the PDF
Figure 6 : Figure 6: Example episode from the dataset.
Figure 6 : Figure 6: Example episode from the dataset.CC-BY · From the PDF
Figure 7 : Figure 7: Architecture diagram of the BC agent.
Figure 7 : Figure 7: Architecture diagram of the BC agent.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 18

  1. C. Bai, X. Zang, Y. Xu, S. Sunkara, A. Rastogi, J. Chen, and B. A. y Arcas. UIBert: Learning generic multimodal representations for UI understanding. In Z. Zhou, editor, Proc. of the 30th International Joint Conference on Artificial Intelligence, IJCAI 2021, pages 1705-1712. ijcai.org, 2021.
  2. P. Banerjee, S. Mahajan, K. Arora, C. Baral, and O. Riva. Lexi: Self-supervised learning of the UI language. In Proc. of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, December 2022.
  3. A. Burns, D. Arsan, S. Agrawal, R. Kumar, K. Saenko, and B. A. Plummer. Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interactive visual environments. CoRR, abs/2104.08560, 2021. · arXiv
  4. J. Chen, C. Chen, Z. Xing, X. Xu, L. Zhu, G. Li, and J. Wang. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. In Proc. of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE '20, pages 322-334, 2020.
  5. J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, and G. Li. Object detection for graphical user interface: Old fashioned or deep learning or a combination? In Proc. of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pages 1202-1214, 2020.
  6. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
  7. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020. · arXiv
  8. I. Gur, U. Rueckert, A. Faust, and D. Hakkani-Tur. Learning to Navigate the Web. In 7th International Conference on Learning Representations (ICLR '19), May 6-9 2019.
  9. Z. He, S. Sunkara, X. Zang, Y. Xu, L. Liu, N. Wichers, G. Schubiner, R. B. Lee, and J. Chen. ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces. In 35th AAAI Conference on Artificial Intelligence, AAAI 2021, pages 5931-5938, 2021.
  10. M. W. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, N. Momchev, D. Sinopalnikov, P. Stańczyk, S. Ramos, A. Raichuk, D. Vincent, L. Hussenot, R. Dadashi, G. Dulac-Arnold, M. Orsini, A. Jacq, J. Ferret, N. Vieillard, S. K. S. Ghasemipour, S. Girgin, O. Pietquin, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Friesen, R. Haroun, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, S. Srinivasan, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020. · arXiv
  11. P. C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, J. Abramson, P. Georgiev, A. Santoro, and T. Lillicrap. A data-driven approach for learning to control computers. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9466-9482. PMLR, 17-23 Jul 2022.
  12. Y. Li, J. He, X. Zhou, Y. Zhang, and J. Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8198-8210. Association for Computational Linguistics, 2020.
  13. E. Z. Liu, K. Guu, P. Pasupat, and P. Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In 6th International Conference on Learning Representations (ICLR '18), 2018.
  14. T. F. Liu, M. Craft, J. Situ, E. Yumer, R. Mech, and R. Kumar. Learning design semantics for mobile apps. In Proc. of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST '18, page 569-579, New York, NY, USA, 2018. Association for Computing Machinery.
  15. A. Richard, H. Kuehne, and J. Gall. Weakly supervised action learning with RNN based fine- to-coarse modeling. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1273-1282, 2017.
  16. T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3135-3144. PMLR, 06-11 Aug 2017.
  17. S. Sunkara, M. Wang, L. Liu, G. Baechler, Y.-C. Hsiao, J. Chen, A. Sharma, and J. W. W. Stout. Towards better semantic understanding of mobile interfaces. In Proc. of the 29th International Conference on Computational Linguistics, pages 5636-5650. International Committee on Computational Linguistics, Oct. 2022.
  18. B. Wang, G. Li, and Y. Li. Enabling conversational interaction with mobile ui using large language models. In Proc. of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23. Association for Computing Machinery, 2023.

Cited by

Metadata

Authors
Rawles, C., Li, A., Rodriguez, D., Riva, O., & Lillicrap, T.
ORCID
Riva, Oriana
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
59708-59728
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2307.10088
arXiv
2307.10088
OpenAlex
W7133225064
Cited by
141
References
0
Citations by year
2026: 58 · 2025: 47 · 2024: 34 · 2023: 2
FWCI
24.99
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Software › Advanced Malware Detection Techniques
  • Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
  • Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
Keywords
Android, mobile device control, large-scale datasets, mobile security, malware analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review