Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 184

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration

Li, Toby Jia-Jun, Azaria, Amos, Myers, Brad A. · 2017 · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

Abstract

SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android's accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps' user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps? UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3025453.3025483

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 39

  1. James Allen , Nathanael Chambers , George Ferguson , 2007 . Plow: A collaborative task learning agent . In Proceedings of the National Conference on Artificial Intelligence, 1514 . James Allen, Nathanael Chambers, George Ferguson, et al. 2007. Plow: A collaborative task learning agent. In Proceedings of the National Conference on Artificial Intelligence, 1514.
  2. V. Antila , J. Polet , A. Lämsä , and J. Liikka . 2012. RoutineMaker: Towards end-user automation of daily routines using smartphones . In 2012 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 399--402 . V. Antila, J. Polet, A. Lämsä, and J. Liikka. 2012. RoutineMaker: Towards end-user automation of daily routines using smartphones. In 2012 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 399--402.
  3. Amos Azaria , Jayant Krishnamurthy , and Tom M. Mitchell . 2016. Instructable intelligent personal agent . In Proc. The 30th AAAI Conference on Artificial Intelligence (AAAI). Amos Azaria, Jayant Krishnamurthy, and Tom M. Mitchell. 2016. Instructable intelligent personal agent. In Proc. The 30th AAAI Conference on Artificial Intelligence (AAAI).
  4. 10.1145/1095034.1095067 · DOI
  5. Aude Billard , Sylvain Calinon , Ruediger Dillmann , and Stefan Schaal . 2008. Robot programming by demonstration . In Springer handbook of robotics. Springer , 1371-- 1394. Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. 2008. Robot programming by demonstration. In Springer handbook of robotics. Springer, 1371-- 1394.
  6. 10.1145/1095034.1095062 · DOI · Paper page
  7. David L. Chen and Raymond J . Mooney . 2011 . Learning to Interpret Natural Language Navigation Instructions from Observations. In AAAI , 1--2. David L. Chen and Raymond J. Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions from Observations. In AAAI, 1--2.
  8. 10.1145/1378773.1378794 · DOI
  9. Yun-Nung Chen , Ming Sun , and Alexander I. Rudnicky . 2015. Matrix factorization with domain knowledge and behavioral patterns for intent modeling . In NIPS Workshop on Machine Learning for SLU and Interaction. Yun-Nung Chen, Ming Sun, and Alexander I. Rudnicky. 2015. Matrix factorization with domain knowledge and behavioral patterns for intent modeling. In NIPS Workshop on Machine Learning for SLU and Interaction.
  10. Allen Cypher and Daniel Conrad Halbert . 1993. Watch what I do: programming by demonstration . MIT press . Allen Cypher and Daniel Conrad Halbert. 1993. Watch what I do: programming by demonstration. MIT press.
  11. 10.1145/168642.168655 · doi:10.1145/168642.168655
  12. 10.1145/192426.192466 · DOI
  13. 10.1145/964442.964461 · DOI · Paper page
  14. 10.1145/1576246.1531372 · DOI
  15. 10.1145/2851581.2892502 · DOI
  16. 10.1145/2736277.2741669 · DOI
  17. 10.1006/jvlc.1996.0011 · DOI · Paper page
  18. 10.1609/aimag.v30i4.2262 · DOI
  19. 10.1145/291080.291104 · DOI
  20. 10.1145/1357054.1357323 · DOI · Paper page
  21. Henry Lieberman. 2001. Your wish is my command: Programming by example. Morgan Kaufmann. Henry Lieberman. 2001. Your wish is my command: Programming by example. Morgan Kaufmann.
  22. 10.1145/176789.176792 · DOI · Paper page
  23. 10.1145/2493190.2493216 · DOI
  24. 10.1145/263407.263515 · DOI
  25. 10.1145/302979.303127 · DOI
  26. 10.1145/259963.260464 · DOI
  27. 10.1145/22627.22349 · DOI · Paper page
  28. 10.1145/78942.78943 · DOI
  29. 10.1145/108844.108903 · DOI
  30. Brad A. Myers and Richard McDaniel . 2001 . Sometimes you need a little intelligence, sometimes you need a lot. Your Wish is My Command : Programming by Example . San Francisco, CA : Morgan Kaufmann Publishers : 45-- 60. Brad A. Myers and Richard McDaniel. 2001. Sometimes you need a little intelligence, sometimes you need a lot. Your Wish is My Command: Programming by Example. San Francisco, CA: Morgan Kaufmann Publishers: 45-- 60.
  31. 10.1145/73660.73672 · doi:10.1145/73660.73672
  32. 10.1177/0278364907079430 · DOI
  33. 10.1109/TSE.2016.2532873 · DOI
  34. 10.1145/2162081.2162087 · DOI
  35. 10.1145/2700648.2811322 · DOI
  36. Ben Shneiderman , Catherine Plaisant , Maxine Cohen , Steven Jacobs , Niklas Elmqvist , and Nicholas Diakopoulos . 2016. Designing the User Interface: Strategies for Effective Human-Computer Interaction . Pearson , Boston . Ben Shneiderman, Catherine Plaisant, Maxine Cohen, Steven Jacobs, Niklas Elmqvist, and Nicholas Diakopoulos. 2016. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, Boston.
  37. Ming Sun , Yun-Nung Chen , and Alexander I. Rudnicky . 2016. HELPR: A Framework to Break the Barrier across Domains in Spoken Dialog Systems . In International Workshop on Spoken Dialog Systems. Ming Sun, Yun-Nung Chen, and Alexander I. Rudnicky. 2016. HELPR: A Framework to Break the Barrier across Domains in Spoken Dialog Systems. In International Workshop on Spoken Dialog Systems.
  38. Jesse Thomason , Shiqi Zhang , Raymond Mooney , and Peter Stone . 2015 . Learning to interpret natural language commands through human-robot dialog . In Proceedings of the Twenty-Fourth international joint conference on Artificial Intelligence (IJCAI). Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth international joint conference on Artificial Intelligence (IJCAI).
  39. 10.1145/2971648.2971696 · DOI

Cited by

See all 184

Metadata

Authors
Li, T. J.-J., Azaria, A., & Myers, B. A.
Venue
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2017-05-02
Pages
6038-6049
OpenAlex
W2610474269
Cited by
184
References
2
FWCI
10.22
Type
conference paper
Language
en
Open access
closed
Funders
  • Samsung (#A017479)
  • Yahoo! (Yahoo! InMind Project)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. https://app.cochat.ai/papers/10.1145/3025453.3025483
The paper
Li, T. J.-J., Azaria, A., & Myers, B. A. (2017). SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3025453.3025483

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.