CoChat

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions

Li, Toby Jia-Jun, Labutov, Igor, Li, Xiaohan Nancy, Zhang, Xiaoyi, Shi, Wenze, Ding, Wanling, Mitchell, Tom M., Myers, Brad A. · 2018 · 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)
Cited by 44

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

A key challenge for generalizing programming-by-demonstration (PBD) scripts is the data description problem - when a user demonstrates performing an action, the system needs to determine features for describing this action and the target object in a way that can reflect the user's intention for the action. However, prior approaches for creating data descriptions in PBD systems have problems with usability, applicability, feasibility, transparency and/or user control. Our APPINITE system introduces a multimodal interface with which users can specify data descriptions verbally using natural language instructions. APPINITE guides users to describe their intentions for the demonstrated actions through mixed-initiative conversations. APPINITE constructs data descriptions for these actions from the natural language instructions. Our evaluation showed that APPINITE is easy-to-use and effective in creating scripts for tasks that would otherwise be difficult to create with prior PBD systems, due to ambiguous data descriptions in demonstrations on GUIs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/vlhcc.2018.8506506

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 51

  1. 10.1145/2462456.2464443 · DOI · Paper page
  2. 10.1145/325737.325845 · DOI · Paper page
  3. 10.1007/978-94-009-7019-9_10 · DOI
  4. 10.1145/2470654.2481301 · DOI · Paper page
  5. thomason, 2015, Learning to Interpret Natural Language Commands Through Human-robot Dialog, Proceedings of the 24th International Conference on Artificial Intelligence, 1923
  6. chen, 2011, Learning to Interpret Natural Language Navigation Instructions from Observations, Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, 859, 10.1609/aaai.v25i1.7974 · DOI · Paper page
  7. 10.1007/1-4020-5386-X_20 · DOI
  8. 10.1109/VLHCC.2005.58 · DOI
  9. 10.1145/2642918.2647395 · DOI
  10. oviatt, 1999, Mutual disambiguation of recognition errors in a multi-model architecture, Proceedings of the SIGCHI conference on Human factors in computing systems, 576, 10.1145/302979.303163 · DOI · Paper page
  11. 10.18653/v1/W15-4661 · doi:10.18653/v1/w15-4661
  12. 10.1145/3025453.3025483 · DOI · Paper page
  13. li, 2018, An End User Development Approach for Failure Handling in Goal-oriented Conversational Agents, Studies in Conversational UX Design, 10.1007/978-3-319-95579-7_6 · DOI
  14. 10.1145/3025171.3025203 · DOI
  15. 10.1609/aimag.v30i4.2262 · DOI · Paper page
  16. 10.1145/800250.807503 · DOI · Paper page
  17. 10.1145/319382.319398 · DOI · Paper page
  18. 10.1177/0278364904049250 · DOI
  19. 10.1109/TIE.2005.858733 · DOI
  20. 10.1145/1518701.1518987 · DOI
  21. 10.1145/2047196.2047226 · DOI · Paper page
  22. 10.1145/1753326.1753554 · DOI · Paper page
  23. 10.1145/3025171.3025176 · DOI
  24. 10.1145/3242587.3242616 · DOI
  25. 10.1145/2984511.2984581 · DOI · Paper page
  26. 10.1006/ijhc.2000.0410 · DOI · Paper page
  27. 10.18653/v1/D17-1161 · DOI
  28. cypher, 1993, Watch What I Do Programming by Demonstration
  29. lieberman, 2001, Your Wish Is My Command Programming by Example
  30. li, 2017, Programming IoT Devices by Demonstration Using Mobile Apps, End-User Development, 3, 10.1007/978-3-319-58735-6_1 · DOI
  31. halbert, 1993, SmallStar: programming by demonstration in the desktop metaphor, Watch What I Do, 103
  32. myers, 2001, Sometimes you need a little intelligence, sometimes you need a lot, Your Wish My Command Program Ex San Franc CA Morgan Kaufmann Publ, 45
  33. 10.1145/274644.274716 · doi:10.1145/274644.274716
  34. myers, 1993, Peridot: creating user interfaces by demonstration, Watch What I Do, 125
  35. 10.1145/1926385.1926423 · DOI · Paper page
  36. 10.1023/A:1025671410623 · DOI · Paper page
  37. 10.1145/3210240.3210339 · DOI · Paper page
  38. 10.1145/302979.303127 · DOI · Paper page
  39. 10.1145/1357054.1357323 · DOI · Paper page
  40. allen, 2007, Plow: A collaborative task learning agent, Proceedings of the National Conference on Artificial Intelligence, 22, 1514
  41. 10.1145/1095034.1095062 · DOI · Paper page
  42. 10.1109/VLHCC.2011.6070381 · DOI
  43. 0, Automate everyday automation for Android LlamaLab
  44. berant, 2013, Semantic parsing on freebase from question-answer pairs, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1533
  45. pruett, 2007, Yahoo! Pipes
  46. 10.1145/2666310.2666493 · doi:10.1145/2666310.2666493
  47. 10.1145/2641580.2641615 · DOI
  48. 10.1162/COLI_a_00127 · DOI · Paper page
  49. auer, 2007, Dbpedia: A nucleus for a web of open data, Semantic Web, 722, 10.1007/978-3-540-76298-0_52 · DOI · Paper page
  50. 0, AccessibilityWindowlnfo | Android Developers
  51. 10.1145/2629489 · DOI · Paper page

Cited by

See all 44

Metadata

Authors
Li, T. J.-J., Labutov, I., Li, X. N., Zhang, X., Shi, W., Ding, W., Mitchell, T. M., & Myers, B. A.
Venue
2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)
Publisher
IEEE
Published
2018-10
Pages
105-114
ISSN
1943-6092 (ISSN-L)
OpenAlex
W2898125335
Cited by
44
References
7
FWCI
3.85
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.