Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has shown that providing AI assistance can improve the accuracy of binary decision tasks. However, the role of AI assistance in more complex data-labeling scenarios with a larger set of labels has not yet been explored. We designed an AI labeling assistant that uses a semi-supervised learning algorithm to predict the most probable labels for each example. We leverage these predictions to provide assistance in two ways: (i) providing a label recommendation and (ii) reducing the labeler’s decision space by focusing their attention on only the most probable labels. We conducted a user study (n=54) to evaluate an AI-assisted interface for data labeling in this context. Our results highlight that the AI assistance improves both labeler accuracy and speed, especially when the labeler finds the correct label in the reduced label space. We discuss findings related to the presentation of AI assistance and design implications for intelligent labeling interfaces.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666
Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743 · cited 1×
“the human not using their judgment to make a decision, can be detrimental to that process [30].”
BackgroundCer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267 · cited 1×
“We encoded each example in the dataset to a 512-dimensional vector using the Universal Sentence Encoder [7].”
MethodThe Magical Mystery Four: How Is Working Memory Capacity Limited, and Why?
Cowan, N. (2010) · Current Directions in Psychological Science · doi:10.1177/0963721409359277 · Cited by 1,255 · cited 1×
“Considering work related to working memory capacity [11, 20], feasibility of visual presentation, and availability of appropriate data, we deemed 21 labels to be sufficiently ‘large’ for our purposes.”
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“Overreliance: “The tendency to over-rely” or automation bias [14] has been investigated in areas such as aviation [25], as well as medicine[14].”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“A common paradigm for supporting and optimizing human decision processes is to leverage human-AI collaboration [6, 35, 40].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 4×
“Joint human-AI decision making outcomes can be improved if user trust towards the AI is calibrated appropriately [42] and when there is an appropriate level of contribution from each party in a human-AI collaborative context [18, 19].”
BackgroundInfluentialHuman-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 2×
“A common paradigm for supporting and optimizing human decision processes is to leverage human-AI collaboration [6, 35, 40].”
BackgroundSelf-citation
Cited by
Gomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Self-citationSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“Prior work has also been dedicated to developing and evaluating interfaces [15, 30, 87, 89] and visualizations [43, 49, 134, 137, 140] aimed at improving human-AI collaboration.”
Extending a Human-AI Collaboration Framework with Dynamism and Sociality
Muller, M., & Weisz, J. (2022) · 2022 Symposium on Human-Computer Interaction for Work · doi:10.1145/3533406.3533407 · Cited by 37
Self-citation
Metadata
- Authors
- Desmond, M., Muller, M., Ashktorab, Z., Dugan, C., Duesterwald, E., Brimijoin, K., Finegan-Dollak, C., Brachman, M., Sharma, A., Joshi, N. N., & Pan, Q.
- Venue
- 26th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2021-04-14
- Pages
- 392-401
- OpenAlex
- W3156222574
- Cited by
- 39
- References
- 17
- FWCI
- 2.84
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface. https://app.cochat.ai/papers/10.1145/3397481.3450698
- The paper
- Desmond, M., Muller, M., Ashktorab, Z., Dugan, C., Duesterwald, E., Brimijoin, K., Finegan-Dollak, C., Brachman, M., Sharma, A., Joshi, N. N., & Pan, Q. (2021). Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface. 26th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3397481.3450698
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.