CoChat
Sign Up
Cited by 207Machine Learning and AlgorithmsUsability and User Interface DesignData Visualization and Analytics

Human model evaluation in interactive supervised learning

Fiebrink, Rebecca, Cook, Perry R., Trueman, Dan·2011·Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
DOI 10.1145/1978942.1978965
Open this paper in CoChat — free account

Abstract

Model evaluation plays a special role in interactive machine learning (IML) systems in which users rely on their assessment of a model's performance in order to determine how to improve it. A better understanding of what model criteria are important to users can therefore inform the design of user interfaces for model evaluation as well as the choice and design of learning algorithms. We present work studying the evaluation practices of end users interactively building supervised learning systems for real-world gesture analysis problems. We examine users' model evaluation criteria, which span conventionally relevant criteria such as accuracy and cost, as well as novel criteria such as unexpectedness. We observed that users employed evaluation techniques---including cross-validation and direct, real-time evaluation---not only to make relevant judgments of algorithms' performance and interactively improve the trained models, but also to learn to provide more effective training data. Furthermore, we observed that evaluation taught users about what types of models were easy or possible to build, and users sometimes used this information to modify the learning problem definition or their plans for using the trained models in practice. We discuss the implications of these findings with regard to the role of generalization accuracy in IML, the design of new algorithms and interfaces, and the scope of potential benefits of incorporating human interaction in the design of supervised learning systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/1978942.1978965

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • IML has been investigated by Fiebrink in the context of expert musicians building new instruments, by creating training examples from real-time demonstrations of human control movements paired with sounds to be produced by those movements.

    “IML has been investigated by Fiebrink [2,3] in the context of expert musicians building new instruments, by creating training examples from real-time demonstrations of human control movements paired with sounds to be produced by those movements.”

    From Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People (2015) · cites this paper as [2]

  • IML-created systems can directly encode aspects of embodied practice and 'feel right' to use.

    “Prior work has shown that interactive machine learning (IML) can help people create custom gesturally-controlled systems quickly and without programming, and that these systems can directly encode aspects of embodied practice and 'feel right' to use [2,3].”

    From Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People (2015) · cites this paper as [2]

  • That prior work demonstrates that IML can improve interface design by facilitating rapid prototyping, exploration of the design space, and encoding of embodied practices into the interface via demonstrated examples.

    “That work demonstrates that IML can improve interface design by facilitating rapid prototyping, exploration of the design space, and encoding of embodied practices into the interface via the demonstrated examples.”

    From Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People (2015) · cites this paper as [2]

  • Prior work on IML focused primarily on highly skilled computer musicians creating custom gestural interfaces for experimental music.

    “This prior work focused primarily on highly skilled computer musicians creating custom gestural interfaces for experimental music.”

    From Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People (2015) · cites this paper as [2]

  • Interactive machine learning (IML) can help people create custom gesturally-controlled systems quickly and without programming.

    “Prior work has shown that interactive machine learning (IML) can help people create custom gesturally-controlled systems quickly and without programming, and that these systems can directly encode aspects of embodied practice and 'feel right' to use [2,3].”

    From Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People (2015) · cites this paper as [2]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. [1]1. Allen, J. M., Asselin, P. K., and Foulds, R. American sign language fingerspelling recognition system. In Proc. of the IEEE 29th Bioengineering Conference (2003).
  2. [4]4. Baker, K., Bhandari, A., and Thotakura, R. An interactive automatic document classification prototype. In Proc. of the Third Workshop on Human-Computer Interaction and Information Retrieval (2009), 30-33.
  3. [10]10. Fiebrink, R., Schedel, M., and Threw, B. Constructing a personalizable gesture-recognizer infrastructure for the K-Bow. In Proc. of the 3rd International Conference on Music and Gesture (MG3) (2010).
  4. [13]13. Fogarty, J., Tan, D., Kapoor, A., and Winder, S. CueFlik: interactive concept learning in image search. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (2008), 29-38.
  5. [19]19. Shilman, M., Tan, D., and Simard, P. CueTIP: A mixed-initiative interface for correcting handwriting errors. In Proc. of the ACM Symposium on User Interface Software and Technology (UIST '06) (2006).
  6. [23]23. Wilson, A., and Shafer, S. XWand: UI for intelligent spaces. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (2003), 545-552.

Cited by

Sorted by citations

Related

Authors
Fiebrink, R., Cook, P. R., & Trueman, D.
Affiliations
Princeton University (US)
ORCID
Fiebrink, Rebecca
Venue
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2011-05-07
Pages
147-156
DOI
10.1145/1978942.1978965
OpenAlex
W1999043044
Cited by
207
References
23
Citations by year
2026: 8 · 2025: 5 · 2024: 12 · 2023: 11 · 2022: 15 · 2021: 21 · 2020: 12 · 2019: 57 · 2018: 13 · 2017: 12 · 2016: 9 · 2015: 12 · 2014: 9 · 2013: 7 · 2012: 3
FWCI
4.99
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • Goldsmiths (University of London) (acceptedVersion)
  • University of the Arts London Research Online (University of the Arts London) (acceptedVersion)
Funders
National Science Foundation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
Keywords
interactive machine learning, model evaluation, gesture analysis, cross-validation, real-time assessment, training data selection, generalization accuracy, human-in-the-loop learning, unexpectedness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review