CoChat
Sign Up
Cited by 18Computational and Text Analysis MethodsSentiment Analysis and Opinion MiningMobile Crowdsensing and Crowdsourcing

Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks

Open access
Schroeder, Hope, Roy, Deb, Kabbara, Jad·2025·Findings of the Association for Computational Linguistics: ACL 2025
DOI 10.18653/v1/2025.findings-acl.1323
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM use in annotation is becoming widespread, and given LLMs' overall promising performance and speed, simply "reviewing" LLM annotations in interpretive tasks can be tempting.In subjective annotation tasks with multiple plausible answers, reviewing LLM outputs can change the label distribution, impacting both the evaluation of LLM performance, and analysis using these labels in a social science task downstream.We conducted a pre-registered experiment with 410 unique annotators and over 7,000 annotations testing three AI assistance conditions against controls, using two models, and two datasets.We find that presenting crowdworkers with LLM-generated annotation suggestions did not make them faster, but did improve their self-reported confidence in the task.More importantly, annotators strongly took the LLM suggestions, significantly changing the label distribution compared to the baseline.When these labels created with LLM assistance are used to evaluate LLM performance, reported model performance significantly increases.We believe our work underlines the importance of understanding the impact of LLM-assisted annotation on subjective, qualitative tasks, on the creation of gold data for training and testing, and on the evaluation of NLP systems on subjective tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2025.findings-acl.1323

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using other VLMs to synthetically generate annotations is a popular approach, but it is likely to perpetuate inaccuracies or biases that the model already has, rather than capturing important nuances.

    “Using other VLMs to synthetically generate annotations is a popular approach [84, 119], but it is likely to perpetuate inaccuracies or biases that the model already has (see distribution shift [110]), rather than capturing important nuances.”

    From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [110]

    The Related Work section says LLMs are used for annotation tasks and widely used in computational social science, supporting the claim that model-generated annotation is an established approach. The paper studies text-focused LLM annotation, however, not the use or popularity of VLMs specifically. Its Discussion also warns that LLM-assisted labels can homogenize judgments, but the citation in the sentence is attached to the approach’s popularity, not that later claim. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 32

  1. Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a rela- tion extraction gold standard. WebSci2013. ACM, 2013(2013).
  2. Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2024. Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways. arXiv preprint. ArXiv:2406.11980 [cs. · arXiv
  3. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2023. Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 37(6):6860-6868. ArXiv:2109.04270 [cs. · DOI · Paper page
  4. Jacques Bughin. 2024. The role of firm ai capabilities in generative ai-pair coding. Journal of Decision Systems, 0(0):1-22. · DOI · Paper page
  5. Alexander S. Choi, Syeda Sabrina Akter, JP Singh, and Antonios Anastasopoulos. 2024. The llm effect: Are humans truly using llms, or are they being influenced by them instead? Preprint, arXiv:2410.04699. · arXiv
  6. Stefano De Paoli. 2023. Can Large Language Mod- els emulate an inductive Thematic Analysis of semi- structured interviews? An exploration and provo- cation on the limits of the approach and the model. arXiv preprint. ArXiv:2305.13014 [cs]. · DOI
  7. Jessica L. Feuston and Jed R. Brubaker. 2021. Putting Tools in Their Place: The Role of Time and Per- spective in Human-AI Collaboration for Qualita- tive Analysis. Proc. ACM Hum.-Comput. Interact., 5(CSCW2):469:1-469:25. · DOI
  8. Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka- Wei Lee, and Simon Perrault. 2023a. CoAIcoder: Examining the Effectiveness of AI-assisted Human- to-Human Collaboration in Qualitative Analysis. ACM Transactions on Computer-Human Interaction, 31(1):6:1-6:38. · DOI
  9. Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka- Wei Lee, and Simon Perrault. 2023b. Coaicoder: Examining the effectiveness of ai-assisted human-to- human collaboration in qualitative analysis. ACM Trans. Comput.-Hum. Interact., 31(1). · DOI
  10. Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Per- rault. 2024. CollabCoder: A Lower-barrier, Rigor- ous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models. In Proceed- ings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24, pages 1-29, New York, NY, USA. Association for Computing Machin- ery. · DOI
  11. Jie Gao, Zhiyao Shu, and Shun Yi Yeo. 2025. Using Large Language Model to Support Flexible and Struc- tural Inductive Qualitative Analysis. arXiv preprint. ArXiv:2501.00775 [cs. · DOI
  12. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. Proceedings of the Na- tional Academy of Sciences, 120(30):e2305016120. ArXiv:2303.15056 [cs]. · DOI · Paper page
  13. Zeyu He, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang. 2024. If in a Crowdsourced Data Annota- tion Pipeline, a GPT-4. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24, pages 1-25, New York, NY, USA. Association for Computing Machinery. · DOI
  14. Abigail Z. Jacobs and Hanna Wallach. 2021. Measure- ment and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, FAccT '21, pages 375-385, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  15. Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T. Hancock, and Mor Naaman. 2019. Ai-mediated com- munication: How the perception that profile text was written by ai affects trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, page 1-13, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  16. Hang Jiang, Xiajie Zhang, Robert Mahari, Daniel Kessler, Eric Ma, Tal August, Irene Li, Alex Pentland, Yoon Kim, Deb Roy, and Jad Kabbara. 2024. Lever- aging large language models for learning complex legal concepts through storytelling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7194-7219, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  17. Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R. Brubaker. 2021. Supporting Serendipity: Op- portunities and Challenges for Human-AI Collabo- ration in Qualitative Analysis. Proc. ACM Hum.- Comput. Interact., 5(CSCW1):94:1-94:23. · DOI · Paper page
  18. Michelle S. Lam, Janice Teoh, James A. Landay, Jef- frey Heer, and Michael S. Bernstein. 2024. Concept Induction: Analyzing Unstructured Text with High- Level Concepts Using LLooM. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1-28, Honolulu HI USA. ACM. · DOI
  19. and Danielle Bragg. 2025. Understanding the LLM- ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review. arXiv preprint. ArXiv:2501.12557 [cs. · DOI
  20. Nicholas Pangakis and Samuel Wolken. 2024. Keep- ing humans in the loop: Human-centered auto- mated annotation with generative ai. arXiv preprint arXiv:2409.09467. · arXiv
  21. Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Jackson Sargent, Apostolos Dedeloudis, and David Jurgens. 2023. POTATO: The Portable Text Annotation Tool. arXiv preprint. ArXiv:2212.08620 [cs]. · DOI
  22. Barbara Plank. 2022. The "Problem" of Human La- bel Variation: On Ground Truth in Data, Modeling and Evaluation. Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 10671-10682. Conference Name: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing Place: Abu Dhabi, United Arab Emirates Publisher: Association for Computational Linguistics. · DOI · Paper page
  23. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. arXiv preprint. ArXiv:2111.07997 [cs]. · DOI
  24. Hope Schroeder, Marianne Aubin Le Quéré, Casey Ran- dazzo, David Mimno, and Sarita Schoenebeck. 2025. Large Language Models in Qualitative Research: Uses, Tensions, and Intentions. In Proceedings of the 2025 CHI Conference on Human Factors in Comput- ing Systems, CHI '25, pages 1-17, New York, NY, USA. Association for Computing Machinery. · DOI
  25. Hope Schroeder, Deb Roy, and Jad Kabbara. 2024. Fora: A corpus and framework for the study of facilitated dialogue. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 13985-14001, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint. ArXiv:2302.13971 [cs]. · DOI · Paper page
  27. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131. · DOI · Paper page
  28. Jenny S. Wang, Samar Haider, Amir Tohidi, Anushkaa Gupta, Yuxuan Zhang, Chris Callison-Burch, David Rothschild, and Duncan J. Watts. 2025. Media Bias Detector: Designing and Implementing a Tool for Real-Time Selection and Framing Bias Analysis in News Coverage. ArXiv:2502.06009 [cs]. · DOI
  29. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want To Reduce La- beling Cost? GPT-3 Can Help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195-4205, Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI
  30. Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. 2023. Primacy Effect of ChatGPT. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 108-115, Singapore. Association for Computational Linguistics. · DOI
  31. Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. 2014. Learning from multiple annotators with varying expertise. Machine Learning, 95(3):291-327. · DOI
  32. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science? arXiv preprint. ArXiv:2305.03514 [cs]. · arXiv

Cited by

Sorted by citations

Related

Authors
Schroeder, H., Roy, D., & Kabbara, J.
Affiliations
Massachusetts Institute of Technology (US)
ORCID
Venue
Findings of the Association for Computational Linguistics: ACL 2025
Publisher
Association for Computational Linguistics
Published
2025
Pages
25771-25795
DOI
10.18653/v1/2025.findings-acl.1323
OpenAlex
W4412944483
Cited by
16
References
0
Citations by year
2026: 13 · 2025: 3
FWCI
28.12
Type
conference paper
Language
en
Open access
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
LLM-assisted annotation, subjective annotation, human-in-the-loop annotation, label distribution shift, LLM evaluation, crowd workers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review