The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Abstract
SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android's accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps' user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps? UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346
Research works printed but not linked: 39
- James Allen , Nathanael Chambers , George Ferguson , 2007 . Plow: A collaborative task learning agent . In Proceedings of the National Conference on Artificial Intelligence, 1514 . James Allen, Nathanael Chambers, George Ferguson, et al. 2007. Plow: A collaborative task learning agent. In Proceedings of the National Conference on Artificial Intelligence, 1514.
- V. Antila , J. Polet , A. Lämsä , and J. Liikka . 2012. RoutineMaker: Towards end-user automation of daily routines using smartphones . In 2012 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 399--402 . V. Antila, J. Polet, A. Lämsä, and J. Liikka. 2012. RoutineMaker: Towards end-user automation of daily routines using smartphones. In 2012 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 399--402.
- Amos Azaria , Jayant Krishnamurthy , and Tom M. Mitchell . 2016. Instructable intelligent personal agent . In Proc. The 30th AAAI Conference on Artificial Intelligence (AAAI). Amos Azaria, Jayant Krishnamurthy, and Tom M. Mitchell. 2016. Instructable intelligent personal agent. In Proc. The 30th AAAI Conference on Artificial Intelligence (AAAI).
- 10.1145/1095034.1095067 · DOI
- Aude Billard , Sylvain Calinon , Ruediger Dillmann , and Stefan Schaal . 2008. Robot programming by demonstration . In Springer handbook of robotics. Springer , 1371-- 1394. Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. 2008. Robot programming by demonstration. In Springer handbook of robotics. Springer, 1371-- 1394.
- 10.1145/1095034.1095062 · DOI · Paper page
- David L. Chen and Raymond J . Mooney . 2011 . Learning to Interpret Natural Language Navigation Instructions from Observations. In AAAI , 1--2. David L. Chen and Raymond J. Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions from Observations. In AAAI, 1--2.
- 10.1145/1378773.1378794 · DOI
- Yun-Nung Chen , Ming Sun , and Alexander I. Rudnicky . 2015. Matrix factorization with domain knowledge and behavioral patterns for intent modeling . In NIPS Workshop on Machine Learning for SLU and Interaction. Yun-Nung Chen, Ming Sun, and Alexander I. Rudnicky. 2015. Matrix factorization with domain knowledge and behavioral patterns for intent modeling. In NIPS Workshop on Machine Learning for SLU and Interaction.
- Allen Cypher and Daniel Conrad Halbert . 1993. Watch what I do: programming by demonstration . MIT press . Allen Cypher and Daniel Conrad Halbert. 1993. Watch what I do: programming by demonstration. MIT press.
- 10.1145/168642.168655 · doi:10.1145/168642.168655
- 10.1145/192426.192466 · DOI
- 10.1145/964442.964461 · DOI · Paper page
- 10.1145/1576246.1531372 · DOI
- 10.1145/2851581.2892502 · DOI
- 10.1145/2736277.2741669 · DOI
- 10.1006/jvlc.1996.0011 · DOI · Paper page
- 10.1609/aimag.v30i4.2262 · DOI
- 10.1145/291080.291104 · DOI
- 10.1145/1357054.1357323 · DOI · Paper page
- Henry Lieberman. 2001. Your wish is my command: Programming by example. Morgan Kaufmann. Henry Lieberman. 2001. Your wish is my command: Programming by example. Morgan Kaufmann.
- 10.1145/176789.176792 · DOI · Paper page
- 10.1145/2493190.2493216 · DOI
- 10.1145/263407.263515 · DOI
- 10.1145/302979.303127 · DOI
- 10.1145/259963.260464 · DOI
- 10.1145/22627.22349 · DOI · Paper page
- 10.1145/78942.78943 · DOI
- 10.1145/108844.108903 · DOI
- Brad A. Myers and Richard McDaniel . 2001 . Sometimes you need a little intelligence, sometimes you need a lot. Your Wish is My Command : Programming by Example . San Francisco, CA : Morgan Kaufmann Publishers : 45-- 60. Brad A. Myers and Richard McDaniel. 2001. Sometimes you need a little intelligence, sometimes you need a lot. Your Wish is My Command: Programming by Example. San Francisco, CA: Morgan Kaufmann Publishers: 45-- 60.
- 10.1145/73660.73672 · doi:10.1145/73660.73672
- 10.1177/0278364907079430 · DOI
- 10.1109/TSE.2016.2532873 · DOI
- 10.1145/2162081.2162087 · DOI
- 10.1145/2700648.2811322 · DOI
- Ben Shneiderman , Catherine Plaisant , Maxine Cohen , Steven Jacobs , Niklas Elmqvist , and Nicholas Diakopoulos . 2016. Designing the User Interface: Strategies for Effective Human-Computer Interaction . Pearson , Boston . Ben Shneiderman, Catherine Plaisant, Maxine Cohen, Steven Jacobs, Niklas Elmqvist, and Nicholas Diakopoulos. 2016. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, Boston.
- Ming Sun , Yun-Nung Chen , and Alexander I. Rudnicky . 2016. HELPR: A Framework to Break the Barrier across Domains in Spoken Dialog Systems . In International Workshop on Spoken Dialog Systems. Ming Sun, Yun-Nung Chen, and Alexander I. Rudnicky. 2016. HELPR: A Framework to Break the Barrier across Domains in Spoken Dialog Systems. In International Workshop on Spoken Dialog Systems.
- Jesse Thomason , Shiqi Zhang , Raymond Mooney , and Peter Stone . 2015 . Learning to interpret natural language commands through human-robot dialog . In Proceedings of the Twenty-Fourth international joint conference on Artificial Intelligence (IJCAI). Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth international joint conference on Artificial Intelligence (IJCAI).
- 10.1145/2971648.2971696 · DOI
Cited by
Zhang, Z., Xu, Y., Wang, Y., Yao, B., Ritchie, D., Wu, T., Yu, M., Wang, D., & Li, T. J.-J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517479 · Cited by 174
Self-citationConversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
Self-citationIn-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119
Exploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40
Metadata
- Authors
- Li, T. J.-J., Azaria, A., & Myers, B. A.
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 6038-6049
- OpenAlex
- W2610474269
- Cited by
- 184
- References
- 2
- FWCI
- 10.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- Samsung (#A017479)
- Yahoo! (Yahoo! InMind Project)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. https://app.cochat.ai/papers/10.1145/3025453.3025483
- The paper
- Li, T. J.-J., Azaria, A., & Myers, B. A. (2017). SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3025453.3025483
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.