AI-Assisted Human Labeling: Batching for Efficiency without Overreliance
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“When adjustments did not directly translate into an updated AI's outcome, users' feedback was considered for future improvement of the model (Ashktorab et al., 2021;Molina and Sundar, 2022;Smith-Renner et al., 2020;Lee et al., 2021).”
From Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review (2025) · cites this paper as Ashktorab et al., 2021
In the citing paper
“Manual labeling after observing AI predictions can be leveraged to identify incorrect intent classifications from textual samples and re-train the models (Ashktorab et al., 2021).”
From Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review (2025) · cites this paper as Ashktorab et al., 2021
Citations
Cites
Advances In Experimental Social Psychology
(2022) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,259
(1996) · Cited by 20,179
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Proceedings of the fifth Berkeley symposium on mathematical statistics and probability.
Kojima, K.-I. (1969) · Europe PMC (PubMed Central) · Cited by 3,648
Construction and Evaluation of a User Experience Questionnaire
Laugwitz, B., Held, T., & Schrepp, M. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89350-9_6 · Cited by 2,114 · cited 2×
“These user experience measures were an average of eight questions asked in the post survey on a 1 to 7 likert scale about the user’s experience on the platforms [34, 56].”
BackgroundMethodHuman-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743 · cited 4×
“This study design allowed us to operationalize overreliance [21, 23, 47, 52] by defining it as the degree to which users agree with a batch recommendation from the AI but are incorrect (e.g., similar to [58]).”
BackgroundInfluentialPlans and situated actions: The problem of human-machine communication
Agre, P. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90078-e · Cited by 1,669
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 3×
“Our technological encounters gradually became “less a matter of pushing buttons and pulling levers” and more about situated interaction, dialogue, and conversation between people and machines [26, 63].”
BackgroundTrust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · cited 1×
“Both of these phenomena have been observed in prior work [36].”
ResultCer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267 · cited 2×
“To achieve this encoding we applied the Universal Sentence Encoder (USE) [10], a transfer-learning model capable of producing semantically rich vector representations from greater-than-word length text.”
Method
Research works printed but not linked: 22
- Mahmoud J Abu Ghali, Abdullah Abu Ayyad, Samy S Abu-Naser, and Mousa Abu Laban. 2018. An intelligent tutoring system for teaching english grammar. (2018).
- Josh Andres, Christine T Wolf, Sergio Cabrero Barros, Erick Oduor, Rahul Nair, Alexander Kjærum, Anders Bech Tharsgaard, and Bo Schwartz Madsen. 2020. Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1--8.
- Yulia Badryzlova, Natalia Shekhtman, Yekaterina Isaeva, and Ruslan Kerimov. 2013. Annotating a Russian corpus of conceptual metaphor: A bottom-up approach. In Proceedings of the First Workshop on Metaphor in NLP. 77--86.
- Victoria A Banks, Katherine L Plant, and Neville A Stanton. 2019. Driving aviation forward; contrasting driving automation and aviation automation. Theoretical issues in ergonomics science, Vol. 20, 3 (2019), 250--264.
- Anthony Brew, Derek Greene, and Pádraig Cunningham. 2010. The interaction between supervised learning and crowdsourcing. In NIPS workshop on computational social science and the wisdom of crowds.
- Andrew Caines, Christian Bentz, Calbert Graham, Tim Polzehl, and Paula Buttery. 2016. Crowdsourcing a multi-lingual speech corpus: Recording, transcription and annotation of the crowdis corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16). 2145--2152.
- Julien Fauqueur, Ashok Thillaisundara, and Theodosia Togia. 2019. Constructing large scale biomedical knowledge bases from scratch with rapid annotation of interpretable patterns. arXiv preprint arXiv:1907.01417 (2019). · arXiv
- 10.5555/1866696.1866709 · doi:10.5555/1866696.1866709
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), Vol. 51, 5 (2018), 1--42.
- Christopher A Miller, Harry Funk, Robert Goldman, John Meisner, and Peggy Wu. 2005. Implications of adaptive vs. adaptable UIs on decision making: Why ?automated adaptiveness" is not always the right answer. In Proceedings of the 1st international conference on augmented cognition. 22--27.
- Michael Muller, Melanie Feinberg, Timothy George, Steven J Jackson, Bonnie E John, Mary Beth Kery, and Samir Passi. 2019. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, W15.
- Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision making, Vol. 5, 5 (2010), 411--419.
- Raja Parasuraman and Dietrich H Manzey. 2010. Complacency and bias in human use of automation: An attentional integration. Human factors, Vol. 52, 3 (2010), 381--410.
- Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors, Vol. 39, 2 (1997), 230--253.
- DJ Patil. 2011. Building data science teams. "O'Reilly Media, Inc.".
- Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. 2020. Interactive machine teaching: a human-centered approach to building machine-learned models. Human-Computer Interaction (2020), 1--39.
- Jack Stewart. 2017. Don't freak over Boeing's self-flying plane-robots already run the skies'. Wired,(online, 6 September 2017) (2017).
- Lucy A Suchman. 1990. What is human-machine interaction. Cognition, computing, and cooperation (1990), 25--55.
- Szilárd Vajda, Yves Rangoni, and Hubert Cecotti. 2015. Semi-automatic ground truth generation using unsupervised clustering and limited manual labeling: Application to handwritten character recognition. Pattern recognition letters, Vol. 58 (2015), 23--28.
- Jason D Williams, Nobal B Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, and Geoff Zweig. 2015. Rapidly scaling dialog systems with interactive learning. In Natural Language Dialog Systems and Intelligent Assistants. Springer, 1--13.
- Antonio Jimeno Yepes, Mariana Neves, and Karin Verspoor. 2013. Brat2BioC: conversion tool between brat and BioC. In Proceedings of the Fourth BioCreative Challenge Evaluation Workshop, Vol. 1. 46--53.
- Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. Webanno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 1--6.
Cited by
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Self-citationGomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
“We then analyzed the application domains where these interactions were studied.”
Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Self-citationChiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 56
Exploring the Use of Personalized AI for Identifying Misinformation on Social Media
Jahanbakhsh, F., Katsis, Y., Wang, D., Popa, L., & Muller, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581219 · Cited by 42
Self-citationChiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making
Lu, Z., Wang, D., & Yin, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653708 · Cited by 39
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
Self-citationExtending a Human-AI Collaboration Framework with Dynamism and Sociality
Muller, M., & Weisz, J. (2022) · 2022 Symposium on Human-Computer Interaction for Work · doi:10.1145/3533406.3533407 · Cited by 37
Self-citationFok, R., Lipka, N., Sun, T., & Siu, A. F. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641969 · Cited by 37
Reza, M., Anastasopoulos, I., Bhandari, S., & Pardos, Z. A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714051 · Cited by 41
Eckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 45
Zhang, S., Xu, H., Jia, Y., Wen, Y., Wang, D., Fu, L., Wang, X., & Zhou, C. (2023) · Geoscience Data Journal · doi:10.1002/gdj3.186 · Cited by 18
Ashktorab, Z., Desmond, M., Pan, Q., Johnson, J., Brachman, M., Dugan, C., Danilevsky, M., & Geyer, W. (2025) · Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work · doi:10.1145/3729176.3729179 · Cited by 10
Self-citationPEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
Zhang, Z., Ning, Z., Xu, C., Tian, Y., & Li, T. J.-J. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606776 · Cited by 19
Ara, Z., Salemi, H., Hong, S.-S., Senarath, Y., Peterson, S., Hughes, A., & Purohit, H. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645214 · Cited by 11
Unveiling Practices of Customer Service Content Curators of Conversational Agents
Candello, H. C. D. S. P., Pinhanez, C. S., Muller, M. J., & Wessel, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555768 · Cited by 11
Self-citationVolkova, A., Hatamian, M., Anapyanova, A., & de Meer, H. (2024) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3677525.3678651 · Cited by 13
AI Simulation by Digital Twins: Systematic Survey of the State of the Art and a Reference Framework
Liu, X., & Dávid, I. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3652620.3688253 · Cited by 12
The Making of Performative Accuracy in AI Training: Precision Labor and Its Consequences
Zhang, B. Z., Yang, T., Miceli, M., Haimson, O. L., & Thomas, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713112 · Cited by 7
Human-centered AI design: developers’ perspectives
Karekezi, P., Moore, G., & Luo, J. (2025) · Journal of Engineering Design · doi:10.1080/09544828.2025.2518907 · Cited by 8
“HCAI defines success in terms of human outcomes and values (Ashktorab et al. 2021; De Cremer and Kasparov 2021).”
Understanding and Shaping Human-Technology Assemblages in the Age of Generative AI
Andrés, J., Danta, C., Bianchi, A., Hong, S., Li, Z., Sandoval, E. B., Martin, C. P., & Cooper, N. (2024) · Designing Interactive Systems Conference · doi:10.1145/3656156.3658403 · Cited by 11
Liu, X., & Dávid, I. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01306-0 · Cited by 28
Radeta, M., Freitas, R., Rodrigues, C., Zuniga, A., Nguyen, N. T., Flores, H., & Nurmi, P. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3649457 · Cited by 14
Data augmentation in human-centric vision
Jiang, W., Zhang, Y., Zheng, S., Liu, S., & Yan, S. (2024) · Vicinagearth. · doi:10.1007/s44336-024-00002-9 · Cited by 10
Keelawat, P. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3607020 · Cited by 10
FOCAL: A Cost-Aware Video Dataset for Active Learning
Kokilepersaud, K., Logan, Y.-Y., Benkert, R., Zhou, C., Prabhushankar, M., AlRegib, G., Corona, E., Singh, K., & Parchami, M. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386242 · Cited by 10
A Scenario-Based Design Pack for Exploring Multimodal Human–GenAI Relations
Andrés, J., Danta, C., Bianchi, A., Farzanfar, S., Fernandez-Nieto, G. M., Becker, A., Capel, T., Liddell, F., Hagemann, S., Cooper, N., et al. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750749 · Cited by 6
Human-Centered AI Design Principles for Generative AI
Weisz, J. D., He, J., & Muller, M. J. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_74 · Cited by 4
Self-citationWang, X., Li, Z., Chen, S., Xing, X., Wan, W., & Li, Q. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712075 · Cited by 3
RelEYEance: Gaze-based Assessment of Users' AI-reliance at Run-time
Wu, Z., Wang, Y., Langer, M., & Feit, A. M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3725841 · Cited by 7
HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains
Zhou, S., Li, B., Chen, X., Tu, Z., Wang, Y., Xiang, Y., & Zhang, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712072 · Cited by 3
Rattanachayabun, K., Jamrat, S., & Samanchuen, T. (2025) · 2025 6th Technology Innovation Management and Engineering Science International Conference (TIMES-iCON) · doi:10.1109/times-icon67125.2025.11488078
Toward an Optimized Human-AI Reviewing Strategy for Contract Inspection
Bancilhon, M., Siu, A. F., Rossi, R. A., & Lipka, N. (2024) · Business, management and economics · doi:10.5772/intechopen.1005255
Enhancing Human-in-the-Loop Learning for Binary Sentiment Word Classification
Martin-Urcelay, B., Rozell, C. J., & Bloch, M. R. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10885942
Effective Model Replacement for Solving Objective Mismatches in Pre-trained Model Compositions
Kharel, A., Baek, K., & Ko, I. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00024
Zlatar, A., & Divers, A. (2025) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0168 · Cited by 1
Julianto, I. T., & Sidqi, M. A. A. (2025) · Journal of Intelligent Systems Technology and Informatics · doi:10.64878/jistics.v1i3.42
Kamande, W., Rheault, C., Gatwiri, M., Duchêne, N., Gachambi, B., & Jauffret, P. (2025) · African Human Computer Interaction Conference (AfriCHI) · doi:10.1145/3757232.3757340 · Cited by 1
The Consensus Trap: Dissecting Subjectivity and the “Ground Truth” Illusion in Data Annotation
Munir, S., Mah, B., Kalsi, K., Kapania, S., Posada, J., Law, E., Wang, D., & Ahmed, S. I. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806510 · Cited by 5
AIANO: Enhancing Information Retrieval with AI-Augmented Annotation
Khattab, S., Bauer, M., Heine, L., Rostalski, T., Kleesiek, J., & Friedrich, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1926-1_18
Supporting Bayesian Reasoning to Mitigate Over-Reliance in AI-Assisted Decision Making
Mikhaylova, D., Turchi, T., Cevolani, G., & Malizia, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30849-8_6
Intelligent Data Annotation: Leveraging AI for Efficient and High-Quality Labelling
Verma, N., Bhardwaj, J., Rana, M., & Yadav, D. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-28594-2_31
Metadata
- Authors
- Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., Duesterwald, E., Dugan, C., Geyer, W., & Reimer, D.
- Affiliations
- IBM (United States) (US)
- IBM Research Australia, Melbourne, VIC, Australia
- IBM Research - Cambridge (United States) (US)
- Independent Researcher, San Jose, CA, USA
- ORCID
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-27
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3158839228
- Cited by
- 45
- References
- 49
- Citations by year
- 2026: 4 · 2025: 17 · 2024: 13 · 2023: 7 · 2022: 4
- FWCI
- 5.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- overreliance on AI, user interface design, Mechanical Turk
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.