Local Decision Pitfalls in Interactive Machine Learning: An Investigation into Feature Selection in Sentiment Analysis
The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index.
Abstract
Tools for Interactive Machine Learning (IML) enable end users to update models in a “rapid, focused, and incremental”—yet local—manner. In this work, we study the question of local decision making in an IML context around feature selection for a sentiment classification task. Specifically, we characterize the utility of interactive feature selection through a combination of human-subjects experiments and computational simulations. We find that, in expectation, interactive modification fails to improve model performance and may hamper generalization due to overfitting. We examine how these trends are affected by the dataset, learning algorithm, and the training set size. Across these factors we observe consistent generalization issues. Our results suggest that rapid iterations with IML systems can be dangerous if they encourage local actions divorced from global context, degrading overall model performance. We conclude by discussing the implications of our feature selection results to the broader area of IML systems and research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1×
“We run the simulations across three commonly used classification algorithms: Logistic Regression, SVM, and random forest, using the implementations within Scikit-Learn [46].”
MethodJudgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · cited 1×
“We also included a maximal random effects structure [8], with a per-subject random inter-cept (capturing individual decision thresholds) and per-subject random slopes for all fixed effects (capturing varied sensitivities to those factors).”
MethodScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 1×
“We run the simulations across three commonly used classification algorithms: Logistic Regression, SVM, and random forest, using the implementations within Scikit-Learn [46].”
MethodThe Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.1080/01621459.1951.10500769 · Cited by 7,210
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Supervised Machine Learning: A Review of Classification Techniques
Kotsiantis, S. B. (2007) · Informatica · Cited by 6,007
Social psychology: handbook of basic principles
(1997) · Choice Reviews Online · doi:10.5860/choice.34-3595 · Cited by 4,895
The Kolmogorov-Smirnov Test for Goodness of Fit
Massey, F. J. (1951) · Journal of the American Statistical Association · doi:10.2307/2280095 · Cited by 3,690 · cited 1×
“In addition, where applicable, we examine the differences between the Δ F1 distribution resulting from different conditions using the Kolmogorov–Smirnov test [40].”
MethodModels of Man--Social and Rational
Solow, R. M., & Simon, H. A. (1958) · The Review of Economics and Statistics · doi:10.2307/1926487 · Cited by 3,146
O'Donoghue, T., & Rabin, M. (1999) · American Economic Review · doi:10.1257/aer.89.1.103 · Cited by 3,121 · cited 1×
“For example, Tversky and Kahneman [65] describe how we tend to rely on one specific piece of information, while O’Donoghue and Rabin [44] note that we over-value immediate rewards.”
BackgroundAn empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780 · cited 1×
“To ensure ecological validity, we used a real sentiment analysis model with random forest classifiers [12, 29].”
MethodSentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining
Baccianella, S., Esuli, A., & Sebastiani, F. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2jktnrykui9s · Cited by 2,666 · cited 2×
“Starting with a word feature list ranked with IG [28], we matched each word with its SentiWordNet polarity score (a continuous score in the range [0 , 1] that we refer as P ) [7].”
MethodEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“This approach can help addresses issues that are not fixable by model iteration alone (e.g., ML fairness [9, 24] and accountability [35]) and may dissuade practitioners from making uninformed or unnecessary structural changes to models.”
BackgroundProceedings of the 16th international joint conference on Artificial Intelligence - IJCAI '99
Colton, S., Bundy, A., & Walsh, T. (1999) · Cited by 2,500
A Synopsis of Linguistic Theory, 1930-1955
Firth, J. R. (1957) · Medical Entomology and Zoology · Cited by 1,675
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · cited 6×
“Interactive machine learning (IML) systems aim to ease the process of training a model by providing tools that support more rapid, focused, and incremental model updates than seen in a traditional machine learning (ML) process [2].”
BackgroundMethodResultInfluentialSnorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Proceedings of the International Conference on Language Resources and Evaluation (LREC)
Wentland, W., Knopp, J., Silberer, C., & Härtung, M. (2008) · Cited by 662
Fails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633 · cited 1×
“Broadly speaking, most IML systems seek to either (1) enable non-programmers to build ML systems [4, 18, 19], T. Wu et al. (2) help end users understand their model failures [1, 3], and (3) improve their models [25, 30, 61].”
BackgroundProceedings of the National Conference on Artificial Intelligence
Jensen, S., Boley, D. L., Gini, M. L., & Schrater, P. (2005) · Innovative Applications of Artificial Intelligence · Cited by 612
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 10×
“As evidenced by the above projects, interactive feature selection is often used in text classification tasks (e.g., [10, 15, 33, 55]).”
BackgroundMethodInfluentialWhy and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
Information gain and a general measure of correlation
KENT, J. T. (1983) · Biometrika · doi:10.1093/biomet/70.1.163 · Cited by 412 · cited 3×
“Starting with a word feature list ranked with IG [28], we matched each word with its SentiWordNet polarity score (a continuous score in the range [0 , 1] that we refer as P ) [7].”
BackgroundMethodTowards better analysis of machine learning models: A visual analytics perspective
Liu, S., Wang, X., Liu, M., & Zhu, J. (2017) · Visual Informatics · doi:10.1016/j.visinf.2017.01.006 · Cited by 397
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
Krause, J., Perer, A., & Ng, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858529 · Cited by 390
The reusable holdout: Preserving validity in adaptive data analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · Science · doi:10.1126/science.aaa9375 · Cited by 307 · cited 1×
“For instance, perhaps IML systems could incorporate a reusable holdout set [17], which uses the idea of differential privacy to safely run tests on a development set multiple times for validation.”
BackgroundUTOPIAN: User-Driven Topic Modeling Based on Interactive Nonnegative Matrix Factorization
Choo, J., Lee, C., Reddy, C. K., & Park, H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.212 · Cited by 262 · cited 3×
“Feature selection, while just one activity supported by IML systems, is provided by many IML systems [10, 15, 33, 55, 60, 61, 63].”
BackgroundMethodModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702509 · Cited by 262 · cited 2×
“For instance, ModelTracker [3] and Squares [52] map the prediction confidence of classifiers to locate false positives/negatives and quantify the level of “incorrect-ness” for users.”
BackgroundiPCA: An Interactive System for PCA‐based Visual Analytics
Jeong, D. H., Ziemkiewicz, C., Fisher, B., Ribarsky, W., & Chang, R. (2009) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01475.x · Cited by 220 · cited 2×
“Users are allowed to either input new instances [37] or manipulate certain features for a given instance [27, 31, 47].”
BackgroundSettles, B. (2011) · Cited by 218
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Chang, J. C., Amershi, S., & Kamar, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3026044 · Cited by 217
Human model evaluation in interactive supervised learning
Fiebrink, R., Cook, P. R., & Trueman, D. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978965 · Cited by 207 · cited 5×
“’s [19] study also report treating a high cross-validation accuracy as reliable evidence that a model was performing well.”
BackgroundInfluentialOverfitting in making comparisons between variable selection methods
ReunanenJuha (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944978 · Cited by 205
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Ren, D., Amershi, S., Lee, B., Suh, J., & Williams, J. D. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598828 · Cited by 200 · cited 3×
“For instance, ModelTracker [3] and Squares [52] map the prediction confidence of classifiers to locate false positives/negatives and quantify the level of “incorrect-ness” for users.”
BackgroundDis-function: Learning distance functions interactively
Brown, E. T., Liu, J., Brodley, C. E., & Chang, R. (2012) · 2012 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2012.6400486 · Cited by 193 · cited 1×
“Across these projects, one highly valued attribute is real-time responses to user feedback (e.g., [6, 11, 25, 54]).”
BackgroundInteracting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184 · cited 10×
“Trivedi et al. [63] report users’ desire for a performance report for the model in each iteration, and Stumpf et al. [60] notice that users paid specific attention to the extent of the accuracy variation.”
BackgroundInfluentialVisual Classifier Training for Text Document Retrieval
Heimerl, F., Koch, S., Bosch, H., & Ertl, T. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.277 · Cited by 177 · cited 4×
“Broadly speaking, most IML systems seek to either (1) enable non-programmers to build ML systems [4, 18, 19], T. Wu et al. (2) help end users understand their model failures [1, 3], and (3) improve their models [25, 30, 61].”
BackgroundInfluentialINFUSE: Interactive Feature Selection for Predictive Modeling of High Dimensional Data
Krause, J., Perer, A., & Bertini, E. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346482 · Cited by 169 · cited 5×
“Broadly speaking, most IML systems seek to either (1) enable non-programmers to build ML systems [4, 18, 19], T. Wu et al. (2) help end users understand their model failures [1, 3], and (3) improve their models [25, 30, 61].”
BackgroundMethodInfluentialIdentifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
Lakkaraju, H., Kamar, E., Caruana, R., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10821 · Cited by 145 · cited 1×
“This approach can help addresses issues that are not fixable by model iteration alone (e.g., ML fairness [9, 24] and accountability [35]) and may dissuade practitioners from making uninformed or unnecessary structural changes to models.”
BackgroundRegroup: interactive machine learning for on-demand group creation in social networks
Amershi, S., Fogarty, J., & Weld, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207680 · Cited by 144
Self-citationHow it works: a field study of non-technical users interacting with an intelligent system
Tullio, J., Dey, A. K., Chalecki, J., & Fogarty, J. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240630 · Cited by 137
Visual explanation of evidence in additive classifiers
Poulin, B., Eisner, R., Szafron, D., Lu, P., Greiner, R., Wishart, D. S., Fyshe, A., Pearcy, B., Macdonell, C., & Anvik, J. (2006) · NPARC · Cited by 131
Investigating statistical machine learning as a tool for software development
Patel, K., Fogarty, J., Landay, J. A., & Harrison, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357160 · Cited by 129 · cited 1×
“However, user studies show that users prefer richer control over ML than simply labeling examples [2, 45, 60].”
BackgroundFlock: Hybrid Crowd-Machine Learning Classifiers
Cheng, J., & Bernstein, M. S. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675214 · Cited by 115 · cited 2×
“These properties enable everyday users to interactively explore the model space through trial-and-error and drive the system toward an intended behavior, hopefully reducing the need for supervision by ML experts [14, 30, 63].”
BackgroundVisual Methods for Analyzing Probabilistic Classification Data
Alsallakh, B., Hanbury, A., Hauser, H., Miksch, S., & Rauber, A. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346660 · Cited by 112 · cited 2×
“Broadly speaking, most IML systems seek to either (1) enable non-programmers to build ML systems [4, 18, 19], T. Wu et al. (2) help end users understand their model failures [1, 3], and (3) improve their models [25, 30, 61].”
BackgroundStructured labeling for facilitating concept evolution in machine learning
Kulesza, T., Amershi, S., Caruana, R., Fisher, D., & Charles, D. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557238 · Cited by 102 · cited 1×
“Creating structured labels [13, 32] for errors in models, or hierarchically structuring their relations based on possible causes, can reflect model strengths and weaknesses on different data segments.”
BackgroundOpinion Mining and Sentiment Analysis: A Survey
Hajmohammadi, M. S., Ibrahim, R. B., & Othman, Z. A. (2012) · INTERNATIONAL JOURNAL OF COMPUTERS & TECHNOLOGY · doi:10.24297/ijct.v2i3c.2717 · Cited by 95
Guiding feature subset selection with an interactive visualization
May, T., Bannach, A., Davey, J., Ruppert, T., & Kohlhammer, J. (2011) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2011.6102448 · Cited by 82 · cited 3×
“May et al. [41] help users filter redundant features.”
BackgroundFeatureInsight: Visual support for error-driven feature ideation in text classification
Brooks, M., Amershi, S., Lee, B., Drucker, S. M., Kapoor, A., & Simard, P. (2015) · 2015 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2015.7347637 · Cited by 80 · cited 9×
“As evidenced by the above projects, interactive feature selection is often used in text classification tasks (e.g., [10, 15, 33, 55]).”
BackgroundInfluentialAn interactive algorithm for asking and incorporating feature feedback into support vector machines
Raghavan, H., & Allan, J. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277758 · Cited by 72 · cited 1×
“For instance, instead of directly modifying the feature space, Raghavan and Allan [48] created additional training data with newly labeled features.”
Why-oriented end-user debugging of naive Bayes text classification
Kulesza, T., Stumpf, S., Wong, W.-K., Burnett, M. M., Perona, S., Ko, A. J., & Oberst, I. (2011) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2030365.2030367 · Cited by 71 · cited 1×
“IML enables two-way interaction between human and machines: On one hand, the system explains to users how the learner is making predictions, usually through visual [38, 41] or textual [34] feedback on model performance [10, 52, 63].”
BackgroundActive Learning with Feedback on Features and Instances
RaghavanHema, MadaniOmid, & JonesRosie (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248608 · Cited by 71
Raghavan, H., Madani, O., & Jones, R. (2005) · Cited by 63
Integrating rich user feedback into intelligent user interfaces
Stumpf, S., Sullivan, E., Fitzhenry, E., Oberst, I., Wong, W., & Burnett, M. (2008) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1378773.1378781 · Cited by 53 · cited 7×
“The most frequently requested [61] and supported manipulations are (1) to specify the feature space (e.g., add or delete features) and (2) to adjust feature weights based on their domain knowledge.”
BackgroundMethodResultInfluentialNLPReViz: an interactive tool for natural language processing on clinical text
Trivedi, G., Pham, P. T., Chapman, W. W., Hwa, R., Wiebe, J., & Hochheiser, H. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx070 · Cited by 44
Examining multiple potential models in end-user interactive concept learning
Amershi, S., Fogarty, J., Kapoor, A., & Tan, D. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753531 · Cited by 41 · cited 1×
“Amershi et al. [5] show that end users regarded leave-one out cross-validation accuracy as a quantity to maximize, and participants in Fiebrink et al.’s [19] study also report treating a high cross-validation accuracy as reliable evidence that a model was performing well.”
BackgroundOverview based example selection in end user interactive concept learning
Amershi, S., Fogarty, J., Kapoor, A., & Tan, D. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622222 · Cited by 41 · cited 2×
“Broadly speaking, most IML systems seek to either (1) enable non-programmers to build ML systems [4, 18, 19], T. Wu et al. (2) help end users understand their model failures [1, 3], and (3) improve their models [25, 30, 61].”
BackgroundEffects of Prior Knowledge on Decisions Made Under Perceptual vs. Categorical Uncertainty
Hansen, K. A., Hillenbrand, S. F., & Ungerleider, L. G. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00163 · Cited by 36 · cited 1×
“Studies have shown that prior knowledge biases decision making under uncertainty [23], visualization perception and interpretation [26], and understanding of vague phrases [22].”
BackgroundDas, S., Moore, T., Wong, W., Stumpf, S., Oberst, I., McIntosh, K., & Burnett, M. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2013.08.003 · Cited by 21 · cited 4×
“Comparing to prior user studies for IML systems,which provided conflicting evidence for IML, our work confirms the observation that human inputs may not improve models [2, 16, 61].”
BackgroundResultInfluentialVarma, P., Iter, D., De, C., & Ré, C. (2017) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3077257.3077263 · Cited by 20
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Simard, P., Chickering, D., Lakshmiratan, A., Charles, D., Bottou, L., Suarez, C. G. J., Grangier, D., Amershi, S., Verwey, J., & Suh, J. (2014) · arXiv · doi:10.48550/arxiv.1409.4814 · Cited by 15 · cited 2×
“[57] propose structured labeling to cope with the evolution of users’ decision boundaries, resulting in more consistently labeled datasets.”
BackgroundAn Interactive Tool for Natural Language Processing on Clinical TextPreprint
Trivedi, G., Pham, P. T., Chapman, W. W., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01890 · Cited by 6 · cited 10×
“These properties enable everyday users to interactively explore the model space through trial-and-error and drive the system toward an intended behavior, hopefully reducing the need for supervision by ML experts [14, 30, 63].”
BackgroundMethodInfluentialThe prior knowledge effect on the processing of vague discourse in Mandarin Chinese
Gong, S.-P., & Ahrens, K. (2011) · Cited by 1
Research works printed but not linked: 7
- Tolga Bolukbasi , Kai-Wei Chang , James Y. Zou , Venkatesh Saligrama , and Adam T . Kalai . 2016 . Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems . 4349--4357. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems. 4349--4357.
- Jessica Hullman . 2013 . How prior knowledge affects the processing of visualized data . In Proceedings of the ACM CHI 2013, Many People Many Eyes Workshop. Jessica Hullman. 2013. How prior knowledge affects the processing of visualized data. In Proceedings of the ACM CHI 2013, Many People Many Eyes Workshop.
- Rensis Likert . 1932. A technique for the measurement of attitudes . In Archives of Psychology . The Science Press , New York . Rensis Likert. 1932. A technique for the measurement of attitudes. In Archives of Psychology. The Science Press, New York.
- 10.5555/2002472.2002491 · doi:10.5555/2002472.2002491
- Vangelis Metsis , Ion Androutsopoulos , and Georgios Paliouras . 2006 . Spam filtering with naive Bayes—which naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS’06) , Vol. 17 . 28--69. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive Bayes—which naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS’06), Vol. 17. 28--69.
- Xin Rong and Eytan Adar . 2016 . Visual tools for debugging neural language models . In Proceedings of ICML Workshop on Visualization for Deep Learning. Xin Rong and Eytan Adar. 2016. Visual tools for debugging neural language models. In Proceedings of ICML Workshop on Visualization for Deep Learning.
- Simone Stumpf , Adrian Bussone , and Dympna O ’sullivan . 2016 . Explanations considered harmful? User interactions with machine learning systems . In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16) . Simone Stumpf, Adrian Bussone, and Dympna O ’sullivan. 2016. Explanations considered harmful? User interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16).
Cited by
Smith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112
Self-citationLiu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
“Similar interactive tools have been used to support data scientists in debugging machine learning models and improving model performance [24, 26, 69].”
MethodExplainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
“For example, some have warned against “local decision pitfalls” [97] of human feedback in iML that overly focuses on modifying a subset of model features, commonly resulting in an overfitted model that fails to generalize.”
BackgroundImproving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87
“For this work, we used behavior descriptions stating the model accuracy for subgroups of the data for which the model performs significantly worse than average [10, 18, 44, 52, 62].”
MethodHuman-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization
Fan, M., Yang, X., Yu, T., Liao, Q. V., & Zhao, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512943 · Cited by 81
Metadata
- Authors
- Wu, T., Weld, D. S., & Heer, J.
- Affiliations
- University of Washington (US)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2019-06-17
- Volume
- 26
- Issue
- 4
- Pages
- 1-27
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2949832227
- Cited by
- 39
- References
- 67
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 3 · 2023: 9 · 2022: 6 · 2021: 9 · 2020: 8
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Funders
- Office of Naval Research (N00014-18-1-2193)
- Washington Research Foundation (Cable Professorship)
- Moore Foundation (Data-Driven Discovery Investigator program)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- interactive machine learning, local decision-making, feature selection, sentiment classification, overfitting, model generalization, human-in-the-loop learning, training data size, learning algorithm comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.