Interactive machine learning
Abstract
Perceptual user interfaces (PUIs) are an important part of ubiquitous computing. Creating such interfaces is difficult because of the image and signal processing knowledge required for creating classifiers. We propose an interactive machine-learning (IML) model that allows users to train, classify/view and correct the classifications. The concept and implementation details of IML are discussed and contrasted with classical machine learning models. Evaluations of two algorithms are also presented. We also briefly describe Image Processing with Crayons (Crayons), which is a tool for creating new camera-based interfaces using a simple painting metaphor. The Crayons tool embodies our notions of interactive machine learning
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The experts' revision of learned knowledge is an important part of the augmented learning process.
“The experts' revision of the learned knowledge is therefore an important part of the augmented learning process (Fails & Olsen, 2003).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Fails & Olsen, 2003
Cites
(1992) · Cited by 23,431 · see citing works
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · see citing works
Seel, N. M. (2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_5195 · Cited by 19,525 · see citing works
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936 · see citing works
Robust real-time object detection.
Viola, P. A. (2001) · Cited by 2,340 · see citing works
Quinlan (2003) · Cited by 997 · see citing works
On the Handling of Continuous-Valued Attributes in Decision Tree Generation
Fayyad, U. M., & Irani, K. B. (1992) · Machine Learning · doi:10.1023/a:1022638503176 · Cited by 746 · see citing works
Implicit human computer interaction through context
Schmidt, A. (2000) · Personal and Ubiquitous Computing · doi:10.1007/bf01324126 · Cited by 639 · see citing works
The basic ideas in neural networks
Rumelhart, D. E., Widrow, B. C., & Lehr, M. A. (1994) · Communications of the ACM · doi:10.1145/175247.175256 · Cited by 462 · see citing works
Gaussian mixture model for human skin color and its applications in image and video databases
Yang, M., & Ahuja, N. (1998) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.333865 · Cited by 299 · see citing works
Starner, T. E., Auxier, J., Ashbrook, D., & Gandy, M. (2002) · International Symposium on Wearable Computers · doi:10.1109/iswc.2000.888469 · Cited by 267 · see citing works
Olsen, D. R., & Nielsen, T. (2001) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/365024.365030 · Cited by 248 · see citing works
Self-citationVIDEOPLACE—an artificial reality
Krueger, M. W., Gionfriddo, T., & Hinrichsen, K. (1985) · ACM SIGCHI Bulletin · doi:10.1145/1165385.317463 · Cited by 242 · see citing works
VIDEOPLACE---an artificial reality
Krueger, M. W., Gionfriddo, T., & Hinrichsen, K. (1985) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/317456.317463 · Cited by 238 · see citing works
Underkoffler, J. S., Ullmer, B., & Ishii, H. (1999) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/311535.311593 · Cited by 169 · see citing works
Underkoffler, J. S., & Ishii, H. (1998) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/274644.274717 · Cited by 166 · see citing works
Conceptual framework for indexing visual information at multiple levels
Jaimes, A., & Chang, S. (1999) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.373443 · Cited by 121 · see citing works
Model-based sonar localisation for mobile robots
Triggs, B. (1994) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(94)90024-8 · Cited by 39 · see citing works
Fails, J. A., & Olsen, D. R. (2002) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/502716.502729 · Cited by 37 · see citing works
Self-citationA hierarchical system for efficient image retrieval
Vailaya, A., Zhong, Y., & Jain, A. K. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.546970 · Cited by 32 · see citing works
Meshcheriakov, R. V. (2024) · Изготовление МЭМС-структур чувствительных элементов датчиков концентрации газа с применением органических щелочей · doi:10.25728/datsys.2024.2.2 · Cited by 25 · see citing works
Automatic selection of visual features and classifiers
Jaimes, A., & Chang, S. (1999) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.373566 · Cited by 17 · see citing works
Integrating Multiple Classifiers In Visual Object Detectors Learned From User Input
Jaimes, A., & Chang, S. (2000) · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755 · see citing works
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · see citing works
“For instance, Crayons demonstrated an early vision of interactive machine learning, allowing non-expert users to train classifiers [30].”
BackgroundPower to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · see citing works
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · see citing works
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · see citing works
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · see citing works
A Review of User Interface Design for Interactive Machine Learning
Dudley, J. J., & Kristensson, P. O. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185517 · Cited by 343 · see citing works
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257 · see citing works
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228 · see citing works
Human model evaluation in interactive supervised learning
Fiebrink, R., Cook, P. R., & Trueman, D. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978965 · Cited by 207 · see citing works
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · see citing works
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184 · see citing works
From explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180 · see citing works
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156 · see citing works
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · see citing works
Discovering Characteristic Actions from On-Body Sensor Data
Minnen, D., Starner, T., Essa, I., & Isbell, C. (2006) · 2006 10th IEEE International Symposium on Wearable Computers · doi:10.1109/iswc.2006.286337 · Cited by 132 · see citing works
BreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102285 · Cited by 125 · see citing works
Understanding and Visualizing Data Iteration in Machine Learning
Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376177 · Cited by 110 · see citing works
Gestalt: integrated support for implementation and analysis in machine learning
Patel, K., Bancroft, N., Drucker, S. M., Fogarty, J., Ko, A. J., & Landay, J. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866038 · Cited by 112 · see citing works
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106 · see citing works
Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows
Xin, D., Wu, E. Y., Lee, D. J.-L., Salehi, N., & Parameswaran, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445306 · Cited by 110 · see citing works
The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality
Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., & Bernstein, M. S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445423 · Cited by 101 · see citing works
Too Late to be Creative? AI-Empowered Tools in Creative Processes
Hwang, A. H.-C. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503549 · Cited by 103 · see citing works
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73 · see citing works
Explaining sentiment analysis results on social media texts through visualization
Jain, R., Kumar, A., Nayyar, A., Dewan, K., Garg, R., Raman, S., & Ganguly, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14432-y · Cited by 68 · see citing works
The Next Generation of Medical Decision Support: A Roadmap Toward Transparent Expert Companions
Bruckert, S., Finzel, B., & Schmid, U. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.507973 · Cited by 66 · see citing works
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415227 · Cited by 57 · see citing works
Katan, S., Grierson, M., & Fiebrink, R. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702474 · Cited by 50 · see citing works
Wu, T., Weld, D. S., & Heer, J. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3319616 · Cited by 39 · see citing works
Ma, S., Wei, Z., Tian, F., Fan, X., Zhang, J., Shen, X., Lin, Z., Huang, J., Měch, R., Samaras, D., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300701 · Cited by 24 · see citing works
Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-69132-7_16 · Cited by 17 · see citing works
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2022) · arXiv · doi:10.48550/arxiv.2211.07819 · Cited by 17
Computing Education Research Today
Fincher, S. A., Tenenberg, J., Dorn, B., Hundhausen, C., McCartney, R., & Murphy, L. (2019) · The Cambridge Handbook of Computing Education Research · doi:10.1017/9781108654555.003 · Cited by 7 · see citing works
Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation
Ko, J., Choi, J., Morales, C., Kim, D., & Ko, M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790929 · Cited by 1 · see citing works
- Authors
- Fails, J. A., & Olsen, D. R.
- Affiliations
- Brigham Young University (US)
- ORCID
- Fails, Jerry Alan
- Venue
- Proceedings of the 8th international conference on Intelligent user interfaces
- Publisher
- ACM
- Published
- 2003-01-12
- Pages
- 39-45
- OpenAlex
- W2003238113
- Cited by
- 633
- References
- 23
- Citations by year
- 2026: 30 · 2025: 81 · 2024: 33 · 2023: 53 · 2022: 35 · 2021: 75 · 2020: 52 · 2019: 73 · 2018: 34 · 2017: 23 · 2016: 30 · 2015: 24 · 2014: 15 · 2013: 6 · 2012: 11
- FWCI
- 2.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- interactive machine learning, perceptual user interfaces, ubiquitous computing, image processing, signal processing, user interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review