Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis
The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index.
Abstract
Tools for Interactive Machine Learning (IML) enable end users to update models in a “rapid, focused, and incremental”—yet local—manner. In this work, we study the question of local decision making in an IML context around feature selection for a sentiment classification task. Specifically, we characterize the utility of interactive feature selection through a combination of human-subjects experiments and computational simulations. We find that, in expectation, interactive modification fails to improve model performance and may hamper generalization due to overfitting. We examine how these trends are affected by the dataset, learning algorithm, and the training set size. Across these factors we observe consistent generalization issues. Our results suggest that rapid iterations with IML systems can be dangerous if they encourage local actions divorced from global context, degrading overall model performance. We conclude by discussing the implications of our feature selection results to the broader area of IML systems and research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1×
“We run the simulations across three commonly used classification algorithms: Logistic Regression, SVM, and random forest, using the implementations within Scikit-Learn [46].”
MethodJudgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · cited 1×
“We also included a maximal random effects structure [8], with a per-subject random inter-cept (capturing individual decision thresholds) and per-subject random slopes for all fixed effects (capturing varied sensitivities to those factors).”
MethodScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 1×
“We run the simulations across three commonly used classification algorithms: Logistic Regression, SVM, and random forest, using the implementations within Scikit-Learn [46].”
MethodThe Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500769 · Cited by 7,210
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Supervised Machine Learning: A Review of Classification Techniques
Kotsiantis, S. B. (2007) · Informatica · Cited by 6,007
Social psychology: handbook of basic principles
(1997) · Choice Reviews Online · doi:10.5860/choice.34-3595 · Cited by 4,895
The Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.2307/2280095 · Cited by 3,690 · cited 1×
“In addition, where applicable, we examine the differences between the Δ F1 distribution resulting from different conditions using the Kolmogorov–Smirnov test [40].”
Method
Research works printed but not linked: 7
- Tolga Bolukbasi , Kai-Wei Chang , James Y. Zou , Venkatesh Saligrama , and Adam T . Kalai . 2016 . Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems . 4349--4357. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems. 4349--4357.
- Jessica Hullman . 2013 . How prior knowledge affects the processing of visualized data . In Proceedings of the ACM CHI 2013, Many People Many Eyes Workshop. Jessica Hullman. 2013. How prior knowledge affects the processing of visualized data. In Proceedings of the ACM CHI 2013, Many People Many Eyes Workshop.
- Rensis Likert . 1932. A technique for the measurement of attitudes . In Archives of Psychology . The Science Press , New York . Rensis Likert. 1932. A technique for the measurement of attitudes. In Archives of Psychology. The Science Press, New York.
- 10.5555/2002472.2002491 · doi:10.5555/2002472.2002491
- Vangelis Metsis , Ion Androutsopoulos , and Georgios Paliouras . 2006 . Spam filtering with naive Bayes—which naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS’06) , Vol. 17 . 28--69. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive Bayes—which naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS’06), Vol. 17. 28--69.
- Xin Rong and Eytan Adar . 2016 . Visual tools for debugging neural language models . In Proceedings of ICML Workshop on Visualization for Deep Learning. Xin Rong and Eytan Adar. 2016. Visual tools for debugging neural language models. In Proceedings of ICML Workshop on Visualization for Deep Learning.
- Simone Stumpf , Adrian Bussone , and Dympna O ’sullivan . 2016 . Explanations considered harmful? User interactions with machine learning systems . In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16) . Simone Stumpf, Adrian Bussone, and Dympna O ’sullivan. 2016. Explanations considered harmful? User interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16).
Cited by
Smith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112
Self-citationLiu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
“Similar interactive tools have been used to support data scientists in debugging machine learning models and improving model performance [24, 26, 69].”
MethodExplainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
“For example, some have warned against “local decision pitfalls” [97] of human feedback in iML that overly focuses on modifying a subset of model features, commonly resulting in an overfitted model that fails to generalize.”
BackgroundImproving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87
“For this work, we used behavior descriptions stating the model accuracy for subgroups of the data for which the model performs significantly worse than average [10, 18, 44, 52, 62].”
MethodHuman-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization
Fan, M., Yang, X., Yu, T., Liao, Q. V., & Zhao, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512943 · Cited by 81
Metadata
- Authors
- Wu, T., Weld, D. S., & Heer, J.
- Affiliations
- University of Washington (US)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2019-06-17
- Volume
- 26
- Issue
- 4
- Pages
- 1-27
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2949832227
- Cited by
- 39
- References
- 67
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 3 · 2023: 9 · 2022: 6 · 2021: 9 · 2020: 8
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Funders
- Office of Naval Research (N00014-18-1-2193)
- Washington Research Foundation (Cable Professorship)
- Moore Foundation (Data-Driven Discovery Investigator program)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- interactive machine learning, local decision-making, feature selection, sentiment classification, overfitting, model generalization, human-in-the-loop learning, training data size, learning algorithm comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis. https://app.cochat.ai/papers/10.1145/3319616
- The paper
- Wu, T., Weld, D. S., & Heer, J. (2019). Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis. ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/3319616
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.