The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
AI-Assisted Human Labeling: Batching for Efficiency without Overreliance
Abstract
Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“When adjustments did not directly translate into an updated AI's outcome, users' feedback was considered for future improvement of the model (Ashktorab et al., 2021;Molina and Sundar, 2022;Smith-Renner et al., 2020;Lee et al., 2021).”
From Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review (2025) · cites this paper as Ashktorab et al., 2021
In the citing paper
“Manual labeling after observing AI predictions can be leveraged to identify incorrect intent classifications from textual samples and re-train the models (Ashktorab et al., 2021).”
From Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review (2025) · cites this paper as Ashktorab et al., 2021
Citations
Cites
Advances In Experimental Social Psychology
(2022) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,259
(1996) · Cited by 20,179
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Proceedings of the fifth Berkeley symposium on mathematical statistics and probability.
Kojima, K.-I. (1969) · Europe PMC (PubMed Central) · Cited by 3,648
Construction and Evaluation of a User Experience Questionnaire
Laugwitz, B., Held, T., & Schrepp, M. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89350-9_6 · Cited by 2,114 · cited 2×
“These user experience measures were an average of eight questions asked in the post survey on a 1 to 7 likert scale about the user’s experience on the platforms [34, 56].”
BackgroundMethodHuman-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743 · cited 4×
“This study design allowed us to operationalize overreliance [21, 23, 47, 52] by defining it as the degree to which users agree with a batch recommendation from the AI but are incorrect (e.g., similar to [58]).”
BackgroundInfluentialPlans and situated actions: The problem of human-machine communication
Agre, P. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90078-e · Cited by 1,669
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 3×
“Our technological encounters gradually became “less a matter of pushing buttons and pulling levers” and more about situated interaction, dialogue, and conversation between people and machines [26, 63].”
BackgroundTrust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · cited 1×
“Both of these phenomena have been observed in prior work [36].”
ResultCer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267 · cited 2×
“To achieve this encoding we applied the Universal Sentence Encoder (USE) [10], a transfer-learning model capable of producing semantically rich vector representations from greater-than-word length text.”
MethodAutomation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 4×
“Prior work on automation bias in clinical decision support systems found several mitigators of overreliance, including position of advice on the screen, confidence levels attached to the automated recommendations and providing information rather than recommending [21].”
BackgroundMethodInfluentialDesign and evaluation of a short version of the User Experience Questionnaire (UEQ-S)
Schrepp, M., Hinderks, A., & Thomaschewski, J. (2017) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2017.09.001 · Cited by 1,058 · cited 2×
“These user experience measures were an average of eight questions asked in the post survey on a 1 to 7 likert scale about the user’s experience on the platforms [34, 56].”
BackgroundMethodPerformance Consequences of Automation-Induced 'Complacency'
Parasuraman, R., Molloy, R., & Singh, I. L. (1993) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0301_1 · Cited by 875 · cited 1×
“Complacency on automated systems has been described as the consistency of the reliability of the automated task in a multi-task environment [46].”
BackgroundFrom Conservation to Crowdsourcing: A Typology of Citizen Science
Wiggins, A., & Crowston, K. (2011) · 2011 44th Hawaii International Conference on System Sciences · doi:10.1109/hicss.2011.207 · Cited by 700 · cited 3×
“In some citizen science projects, the administrators (professional scientists) sometimes check the quality of their volunteer labelers by inserting test items which have known ground-truth answers into the labelers’ workflow [69].”
BackgroundKoehn, P., & Hoang, H. T. (2007) · The Association for Computational Linguistics · Cited by 673
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“A number of CSCW papers in recent years have addressed questions of human-AI collaboration, for example in medical decision-making [8], data science work [66], or IT infrastructure design practices [72].”
BackgroundUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“In a recent study, user trust wasmeasured based on how likely an individual was to follow the recommendations given by an AI [75].”
BackgroundSnorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
Machine Heuristic: When We Trust Computers More than Humans with Our Personal Information
Sundar, S. S., & Kim, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300768 · Cited by 347 · cited 1×
“Many instruments that measure trust ask how likely an individual is to follow the recommendations given by that system [12, 31, 64].”
BackgroundA Review of User Interface Design for Interactive Machine Learning
Dudley, J. J., & Kristensson, P. O. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185517 · Cited by 343 · cited 2×
“Interactive machine teaching is a growing area that sees the ML model training process through a human-centered lens [15].”
BackgroundHuman-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 2×
“A number of CSCW papers in recent years have addressed questions of human-AI collaboration, for example in medical decision-making [8], data science work [66], or IT infrastructure design practices [72].”
BackgroundSelf-citationExplaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 1×
“Many instruments that measure trust ask how likely an individual is to follow the recommendations given by that system [12, 31, 64].”
BackgroundAgency plus automation: Designing artificial intelligence into interactive systems
Heer, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807184115 · Cited by 279 · cited 3×
“This study design allowed us to operationalize overreliance [21, 23, 47, 52] by defining it as the degree to which users agree with a batch recommendation from the AI but are incorrect (e.”
BackgroundMethodHow do Data Science Workers Collaborate? Roles, Workflows, and Tools
Zhang, A. X., Muller, M., & Wang, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392826 · Cited by 268 · cited 3×
“After the labels have been added to data records, then the datset itself is passed along to other roles in the complex hierarchy of data science teams [28] and their work-practices [76].”
BackgroundInfluentialSelf-citationArtificial Intelligence and the Future of Humans
Ovidius University of Constanta, D. O. T., & Ciocan, T.-C. (2023) · DIALOGO · doi:10.51917/dialogo.2023.10.1.15 · Cited by 243 · cited 1×
“While a recent survey found AI-decision-making systems to be unfavorable with people [2], other studies have found that in many domains individuals believe in the inevitability of human-AI collaboration and were optimistic about the potential in data science [66], music [11], pedagogical tools for health literacy [61]…”
The emerging role of data scientists on software development teams
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884783 · Cited by 231 · cited 2×
“Teams frequently engage in a division-of-labor according to specialized roles and skills [19, 30, 40, 68, 69].”
BackgroundJakesch, M., French, M., Ma, X., Hancock, J. T., & Naaman, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300469 · Cited by 190 · cited 10×
“After the labeling task was complete, participants also answered demographic questions and questions about prior experience with Mechanical Turk as well as free-form questions explaining their impressions of the labeling system.”
BackgroundResultInfluentialLet Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156 · cited 1×
“Many instruments that measure trust ask how likely an individual is to follow the recommendations given by that system [12, 31, 64].”
BackgroundWorker types and personality traits in crowdsourcing relevance labels
Kazai, G., Kamps, J., & Milić-Frayling, N. (2011) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2063576.2063860 · Cited by 154 · cited 1×
“Crowdsourcing platforms offer an abundance of relatively cheap labor and are widespread for labeling tasks [4, 29, 37].”
BackgroundMixed-Initiative Creative Interfaces
Deterding, S., Hook, J., Fiebrink, R., Gillies, M., Gow, J., Akten, M., Smith, G., Liapis, A., & Compton, K. (2017) · Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/3027063.3027072 · Cited by 117 · cited 2×
“The growing prevalence of contemporary AI in a wide array of everyday systems has called our attention to the nature and experience of interactions between humans and intelligent machines [14].”
BackgroundUnderstanding and Visualizing Data Iteration in Machine Learning
Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376177 · Cited by 110 · cited 1×
“In a simple sense, any labeling or annotating task involves a human teaching a machine-learning model about the ground truth of the data [24, 49].”
BackgroundOn the Ethics of Crowdsourced Research
Williamson, V. S. (2016) · PS Political Science & Politics · doi:10.1017/s104909651500116x · Cited by 106 · cited 1×
“We must acknowledge that many crowdworkers on these platforms are not one-time users, but also individuals who earn their income through these platforms [71].”
BackgroundMitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102 · cited 2×
“Getting quality work from crowdworkers is a challenge, as demonstrated by prior work that investigated ways to get quality data from crowdworkers through aptitude tests and training in qualitative coding, showing that in-person training can improve quality of data collected on crowdsourcing platforms [42].”
BackgroundStatistical quality estimation for general crowdsourcing tasks
Baba, Y., & Kashima, H. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2487600 · Cited by 86 · cited 1×
“Crowdsourcing platforms offer an abundance of relatively cheap labor and are widespread for labeling tasks [4, 29, 37].”
BackgroundWeidele, D. K. I., Weisz, J. D., Oduor, E., Muller, M. J., Andrés, J., Gray, A., & Wang, D. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377538 · Cited by 74 · cited 1×
“Recent work in the HCI and CSCW community investigates how users try to understand and form impressions of AI systems and outcomes in human-AI collaborative settings [3, 67].”
BackgroundSelf-citationDeveloping a conceptual model of virtual organisations for citizen science
Wiggins, A., & Crowston, K. (2010) · International Journal of Organisational Design and Engineering · doi:10.1504/ijode.2010.035191 · Cited by 73
Crowdsourcing High Quality Labels with a Tight Budget
Li, Q., Ma, F., Gao, J., Su, L., & Quinn, C. J. (2016) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2835776.2835797 · Cited by 54 · cited 1×
“Crowdsourcing platforms offer an abundance of relatively cheap labor and are widespread for labeling tasks [4, 29, 37].”
BackgroundEvaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices
Wolf, C., & Blomberg, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359245 · Cited by 48 · cited 1×
“A number of CSCW papers in recent years have addressed questions of human-AI collaboration, for example in medical decision-making [8], data science work [66], or IT infrastructure design practices [72].”
BackgroundSelf-citationMonotasking or Multitasking: Designing for Crowdworkers' Preferences
Lascau, L., Gould, S. J. J., Cox, A. L., Karmannaya, E., & Brumby, D. P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300649 · Cited by 47 · cited 1×
“Prior work has shown that Mechanical Turkers multi-task, often taking breaks between tasks [33] or doing multiple HITs (Human Intelligent Tasks) simultaneously.”
Crowdsourcing the Perception of Machine Teaching
Hong, J., Lee, K., Xu, J., & Kacorri, H. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376428 · Cited by 39 · cited 1×
“Hong et al [25] used a machine teaching exercise (e.g., mTurk users uploaded photos to teach the model various categories) to understand how human teachers understand and reflect on their experiences with ML.”
BackgroundRobust semi-automatic head pose labeling for real-world face video sequences
Demirkus, M., Clark, J. J., & Arbel, T. (2013) · Multimedia Tools and Applications · doi:10.1007/s11042-012-1352-1 · Cited by 35 · cited 1×
“[13] developed a semi-automatic framework for labeling temporal head poses in real-world video sequences, a task that is both time-intensive and expensive.”
BackgroundFort, K. (2016) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 32
Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376290 · Cited by 28 · cited 1×
“Subject Matter Expert labeling frameworks also exist [54], but there the labor force is small and likely extremely expensive.”
BackgroundSchmidt, F. A. (2017) · De Gruyter eBooks · doi:10.1515/9783035610673 · Cited by 10 · cited 1×
“In the age of crowdwork-ing platforms, scholars have categorized two classes of Mechanical Turk Stakeholders: one that outsources small repetitive tasks and one that is on the receiving end doing these mundane tasks [55].”
BackgroundChamberlain, A., Malizia, A., & Roure, D. D. (2017) · Repository@Nottingham (University of Nottingham) · Cited by 4
The Science of Managing Data Science
Matsudaira, K. (2015) · Queue · doi:10.1145/2756506.2767971 · Cited by 3 · cited 2×
“Teams frequently engage in a division-of-labor according to specialized roles and skills [19, 30, 40, 68, 69].”
BackgroundHealthy Data: Policy Solutions for Big Data and AI Innovation in Health
Stinson, C. (2018) · TSpace (University of Toronto) · Cited by 2
Research works printed but not linked: 23
- Mahmoud J Abu Ghali, Abdullah Abu Ayyad, Samy S Abu-Naser, and Mousa Abu Laban. 2018. An intelligent tutoring system for teaching english grammar. (2018).
- Josh Andres, Christine T Wolf, Sergio Cabrero Barros, Erick Oduor, Rahul Nair, Alexander Kjærum, Anders Bech Tharsgaard, and Bo Schwartz Madsen. 2020. Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1--8.
- Yulia Badryzlova, Natalia Shekhtman, Yekaterina Isaeva, and Ruslan Kerimov. 2013. Annotating a Russian corpus of conceptual metaphor: A bottom-up approach. In Proceedings of the First Workshop on Metaphor in NLP. 77--86.
- Victoria A Banks, Katherine L Plant, and Neville A Stanton. 2019. Driving aviation forward; contrasting driving automation and aviation automation. Theoretical issues in ergonomics science, Vol. 20, 3 (2019), 250--264.
- Anthony Brew, Derek Greene, and Pádraig Cunningham. 2010. The interaction between supervised learning and crowdsourcing. In NIPS workshop on computational social science and the wisdom of crowds.
- Andrew Caines, Christian Bentz, Calbert Graham, Tim Polzehl, and Paula Buttery. 2016. Crowdsourcing a multi-lingual speech corpus: Recording, transcription and annotation of the crowdis corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16). 2145--2152.
- Julien Fauqueur, Ashok Thillaisundara, and Theodosia Togia. 2019. Constructing large scale biomedical knowledge bases from scratch with rapid annotation of interpretable patterns. arXiv preprint arXiv:1907.01417 (2019). · arXiv
- 10.5555/1866696.1866709 · doi:10.5555/1866696.1866709
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), Vol. 51, 5 (2018), 1--42.
- Christopher A Miller, Harry Funk, Robert Goldman, John Meisner, and Peggy Wu. 2005. Implications of adaptive vs. adaptable UIs on decision making: Why ?automated adaptiveness" is not always the right answer. In Proceedings of the 1st international conference on augmented cognition. 22--27.
- Michael Muller, Melanie Feinberg, Timothy George, Steven J Jackson, Bonnie E John, Mary Beth Kery, and Samir Passi. 2019. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, W15.
- Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision making, Vol. 5, 5 (2010), 411--419.
- Raja Parasuraman and Dietrich H Manzey. 2010. Complacency and bias in human use of automation: An attentional integration. Human factors, Vol. 52, 3 (2010), 381--410.
- Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors, Vol. 39, 2 (1997), 230--253.
- DJ Patil. 2011. Building data science teams. "O'Reilly Media, Inc.".
- Claudio Pinhanez. 2019. Machine Teaching by Domain Experts: Towards More Humane, Inclusive, and Intelligent Machine Learning Systems. arXiv preprint arXiv:1908.08931 (2019). · arXiv
- Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. 2020. Interactive machine teaching: a human-centered approach to building machine-learned models. Human-Computer Interaction (2020), 1--39.
- Jack Stewart. 2017. Don't freak over Boeing's self-flying plane-robots already run the skies'. Wired,(online, 6 September 2017) (2017).
- Lucy A Suchman. 1990. What is human-machine interaction. Cognition, computing, and cooperation (1990), 25--55.
- Szilárd Vajda, Yves Rangoni, and Hubert Cecotti. 2015. Semi-automatic ground truth generation using unsupervised clustering and limited manual labeling: Application to handwritten character recognition. Pattern recognition letters, Vol. 58 (2015), 23--28.
- Jason D Williams, Nobal B Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, and Geoff Zweig. 2015. Rapidly scaling dialog systems with interactive learning. In Natural Language Dialog Systems and Intelligent Assistants. Springer, 1--13.
- Antonio Jimeno Yepes, Mariana Neves, and Karin Verspoor. 2013. Brat2BioC: conversion tool between brat and BioC. In Proceedings of the Fourth BioCreative Challenge Evaluation Workshop, Vol. 1. 46--53.
- Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. Webanno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 1--6.
Cited by
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Self-citationGomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
“We then analyzed the application domains where these interactions were studied.”
Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
Self-citationChiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 97
Metadata
- Authors
- Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., Duesterwald, E., Dugan, C., Geyer, W., & Reimer, D.
- Affiliations
- IBM (United States) (US)
- IBM Research Australia, Melbourne, VIC, Australia
- IBM Research - Cambridge (United States) (US)
- Independent Researcher, San Jose, CA, USA
- ORCID
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-27
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3158839228
- Cited by
- 45
- References
- 49
- Citations by year
- 2026: 4 · 2025: 17 · 2024: 13 · 2023: 7 · 2022: 4
- FWCI
- 5.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- overreliance on AI, user interface design, Mechanical Turk
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.