Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.

AI-Assisted Human Labeling: Batching for Efficiency without Overreliance

Abstract

Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.

Overview

Integrity signals

Registry

Crossref · 10.1145/3449163

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 49 references

Research works printed but not linked: 23

  1. Mahmoud J Abu Ghali, Abdullah Abu Ayyad, Samy S Abu-Naser, and Mousa Abu Laban. 2018. An intelligent tutoring system for teaching english grammar. (2018).
  2. Josh Andres, Christine T Wolf, Sergio Cabrero Barros, Erick Oduor, Rahul Nair, Alexander Kjærum, Anders Bech Tharsgaard, and Bo Schwartz Madsen. 2020. Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1--8.
  3. Yulia Badryzlova, Natalia Shekhtman, Yekaterina Isaeva, and Ruslan Kerimov. 2013. Annotating a Russian corpus of conceptual metaphor: A bottom-up approach. In Proceedings of the First Workshop on Metaphor in NLP. 77--86.
  4. Victoria A Banks, Katherine L Plant, and Neville A Stanton. 2019. Driving aviation forward; contrasting driving automation and aviation automation. Theoretical issues in ergonomics science, Vol. 20, 3 (2019), 250--264.
  5. Anthony Brew, Derek Greene, and Pádraig Cunningham. 2010. The interaction between supervised learning and crowdsourcing. In NIPS workshop on computational social science and the wisdom of crowds.
  6. Andrew Caines, Christian Bentz, Calbert Graham, Tim Polzehl, and Paula Buttery. 2016. Crowdsourcing a multi-lingual speech corpus: Recording, transcription and annotation of the crowdis corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16). 2145--2152.
  7. Julien Fauqueur, Ashok Thillaisundara, and Theodosia Togia. 2019. Constructing large scale biomedical knowledge bases from scratch with rapid annotation of interpretable patterns. arXiv preprint arXiv:1907.01417 (2019). · arXiv
  8. 10.5555/1866696.1866709 · doi:10.5555/1866696.1866709
  9. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), Vol. 51, 5 (2018), 1--42.
  10. Christopher A Miller, Harry Funk, Robert Goldman, John Meisner, and Peggy Wu. 2005. Implications of adaptive vs. adaptable UIs on decision making: Why ?automated adaptiveness" is not always the right answer. In Proceedings of the 1st international conference on augmented cognition. 22--27.
  11. Michael Muller, Melanie Feinberg, Timothy George, Steven J Jackson, Bonnie E John, Mary Beth Kery, and Samir Passi. 2019. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, W15.
  12. Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision making, Vol. 5, 5 (2010), 411--419.
  13. Raja Parasuraman and Dietrich H Manzey. 2010. Complacency and bias in human use of automation: An attentional integration. Human factors, Vol. 52, 3 (2010), 381--410.
  14. Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors, Vol. 39, 2 (1997), 230--253.
  15. DJ Patil. 2011. Building data science teams. "O'Reilly Media, Inc.".
  16. Claudio Pinhanez. 2019. Machine Teaching by Domain Experts: Towards More Humane, Inclusive, and Intelligent Machine Learning Systems. arXiv preprint arXiv:1908.08931 (2019). · arXiv
  17. Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. 2020. Interactive machine teaching: a human-centered approach to building machine-learned models. Human-Computer Interaction (2020), 1--39.
  18. Jack Stewart. 2017. Don't freak over Boeing's self-flying plane-robots already run the skies'. Wired,(online, 6 September 2017) (2017).
  19. Lucy A Suchman. 1990. What is human-machine interaction. Cognition, computing, and cooperation (1990), 25--55.
  20. Szilárd Vajda, Yves Rangoni, and Hubert Cecotti. 2015. Semi-automatic ground truth generation using unsupervised clustering and limited manual labeling: Application to handwritten character recognition. Pattern recognition letters, Vol. 58 (2015), 23--28.
  21. Jason D Williams, Nobal B Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, and Geoff Zweig. 2015. Rapidly scaling dialog systems with interactive learning. In Natural Language Dialog Systems and Intelligent Assistants. Springer, 1--13.
  22. Antonio Jimeno Yepes, Mariana Neves, and Karin Verspoor. 2013. Brat2BioC: conversion tool between brat and BioC. In Proceedings of the Fourth BioCreative Challenge Evaluation Workshop, Vol. 1. 46--53.
  23. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. Webanno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 1--6.

Cited by

See all 45

Metadata

Authors
Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., Duesterwald, E., Dugan, C., Geyer, W., & Reimer, D.
Affiliations
  • IBM (United States) (US)
  • IBM Research Australia, Melbourne, VIC, Australia
  • IBM Research - Cambridge (United States) (US)
  • Independent Researcher, San Jose, CA, USA
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2021-04-13
Volume
5
Issue
CSCW1
Pages
1-27
ISSN
2573-0142 (ISSN-L)
OpenAlex
W3158839228
Cited by
45
References
49
Citations by year
2026: 4 · 2025: 17 · 2024: 13 · 2023: 7 · 2022: 4
FWCI
5.08
Type
journal article
Language
en
Open access
closed
Locations
Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
overreliance on AI, user interface design, Mechanical Turk

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. https://app.cochat.ai/papers/10.1145/3449163
The paper
Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., Duesterwald, E., Dugan, C., Geyer, W., & Reimer, D. (2021). AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3449163

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.