CoChat

Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis

Wu, Tongshuang, Weld, Daniel S., Heer, Jeffrey · 2019 · ACM Transactions on Computer-Human Interaction

The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index.

Abstract

Tools for Interactive Machine Learning (IML) enable end users to update models in a “rapid, focused, and incremental”—yet local—manner. In this work, we study the question of local decision making in an IML context around feature selection for a sentiment classification task. Specifically, we characterize the utility of interactive feature selection through a combination of human-subjects experiments and computational simulations. We find that, in expectation, interactive modification fails to improve model performance and may hamper generalization due to overfitting. We examine how these trends are affected by the dataset, learning algorithm, and the training set size. Across these factors we observe consistent generalization issues. Our results suggest that rapid iterations with IML systems can be dangerous if they encourage local actions divorced from global context, degrading overall model performance. We conclude by discussing the implications of our feature selection results to the broader area of IML systems and research.

Overview

Integrity signals

Registry

Crossref · 10.1145/3319616

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 66 references

Research works printed but not linked: 7

  1. Tolga Bolukbasi , Kai-Wei Chang , James Y. Zou , Venkatesh Saligrama , and Adam T . Kalai . 2016 . Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems . 4349--4357. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems. 4349--4357.
  2. Jessica Hullman . 2013 . How prior knowledge affects the processing of visualized data . In Proceedings of the ACM CHI 2013, Many People Many Eyes Workshop. Jessica Hullman. 2013. How prior knowledge affects the processing of visualized data. In Proceedings of the ACM CHI 2013, Many People Many Eyes Workshop.
  3. Rensis Likert . 1932. A technique for the measurement of attitudes . In Archives of Psychology . The Science Press , New York . Rensis Likert. 1932. A technique for the measurement of attitudes. In Archives of Psychology. The Science Press, New York.
  4. 10.5555/2002472.2002491 · doi:10.5555/2002472.2002491
  5. Vangelis Metsis , Ion Androutsopoulos , and Georgios Paliouras . 2006 . Spam filtering with naive Bayes—which naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS’06) , Vol. 17 . 28--69. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive Bayes—which naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS’06), Vol. 17. 28--69.
  6. Xin Rong and Eytan Adar . 2016 . Visual tools for debugging neural language models . In Proceedings of ICML Workshop on Visualization for Deep Learning. Xin Rong and Eytan Adar. 2016. Visual tools for debugging neural language models. In Proceedings of ICML Workshop on Visualization for Deep Learning.
  7. Simone Stumpf , Adrian Bussone , and Dympna O ’sullivan . 2016 . Explanations considered harmful? User interactions with machine learning systems . In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16) . Simone Stumpf, Adrian Bussone, and Dympna O ’sullivan. 2016. Explanations considered harmful? User interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16).

Cited by

See all 39

Metadata

Authors
Wu, T., Weld, D. S., & Heer, J.
Affiliations
University of Washington (US)
Venue
ACM Transactions on Computer-Human Interaction
Published
2019-06-17
Volume
26
Issue
4
Pages
1-27
ISSN
  • 1073-0516 (ISSN-L)
  • 1557-7325
OpenAlex
W2949832227
Cited by
39
References
67
Citations by year
2026: 1 · 2025: 3 · 2024: 3 · 2023: 9 · 2022: 6 · 2021: 9 · 2020: 8
FWCI
2.62
Type
journal article
Language
en
Open access
closed
Locations
ACM Transactions on Computer-Human Interaction (publishedVersion)
Funders
  • Google
  • Office of Naval Research (N00014-18-1-2193)
  • Washington Research Foundation (Cable Professorship)
  • Moore Foundation (Data-Driven Discovery Investigator program)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
interactive machine learning, local decision-making, feature selection, sentiment classification, overfitting, model generalization, human-in-the-loop learning, training data size, learning algorithm comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis. https://app.cochat.ai/papers/10.1145/3319616
The paper
Wu, T., Weld, D. S., & Heer, J. (2019). Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis. ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/3319616

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.