CoChat
Sign Up
Cited by 27Computational and Text Analysis MethodsSentiment Analysis and Opinion MiningTopic Modeling

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

Open access
Pangakis, Nick, Wolken, Sam·2025·Proceedings of the International AAAI Conference on Web and Social Media
DOI 10.1609/icwsm.v19i1.35883
Open this paper in CoChat — free accountOpen PDF

Abstract

Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimization strategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essential for responsible evaluation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/icwsm.v19i1.35883

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models can generate candidate annotations or explanations that human annotators then verify, correct, or refine.

    “Large language models can generate candidate annotations or explanations that human annotators then verify, correct, or refine [34].”

    From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [34]

    The Introduction and Replication Procedures describe LLM-generated annotations evaluated against human labels, and the discussion of uncertain labels identifies cases that may need human inspection. The paper does not establish that annotators directly correct or refine individual LLM outputs, or that the model generates explanations for them to review. No page numbers were available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The 200 job posts represent over 5% of the database and are in line with recommendations for LLM-classification validation from prior studies.

    “This represents over 5% of the database and is in line with recommendations for LLM-classification validation from prior studies [53].”

    From The quantum technology job market: data driven analysis of 3641 job posts (2026) · cites this paper as [53]

    The Discussion (page unavailable) recommends task-by-task human validation of LLM annotations. The Replication Procedures describe validation against 1,000 held-out human-labeled samples, but the paper does not recommend a 200-post sample or a 5% threshold. The cited recommendation therefore supports validation in general, not the claim that this particular sample size is in line with it. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The validation framework was inspired by the works of Shah [52] and Pangakis et al. [53] and customized to the project's objectives and resource constraints.

    “The validation framework utilized in this study is inspired by the works of Shah [52] and Pangakis et al., [53] and has been customized to meet the specific objectives and resource constraints of our project”

    From The quantum technology job market: data driven analysis of 3641 job posts (2026) · cites this paper as [53]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [5]5. Curran Associates, Inc. Card, D.; Chang, S.; Becker, C.; Mendelsohn, J.; Voigt, R.; Boustan, L.; Abramitzky, R.; and Jurafsky, D. 2022. Com- putational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sci- ences of the United States of America, 31.
  2. [8]8. Cusimano, C.; and Goodwin, G. P. 2020. People judge oth- ers to have more voluntary control over beliefs than they themselves do. Journal of Personality and Social Psychol- ogy, 119.
  3. [9]9. Davani, A. M.; Díaz, M.; and Prabhakaran, V. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10: 92-110.
  4. [10]10. Devlin, J.; Chang, M.-W.; Le, K.; and Toutanova, K. 2019. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171-4186.
  5. [13]13. Espinosa, L.; and Salathé, M. 2024. Use of large lan- guage models as a scalable approach to understanding public health discourse. medRxiv, 2024-02.
  6. [15]15. Felkner, V.; Thompson, J.; and May, J. 2024. GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14104-14115. Bangkok, Thailand: Associa- tion for Computational Linguistics.
  7. [16]16. Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.; Wallach, H.; Iii, H. D.; and Crawford, K. 2021. Datasheets for datasets. Communications of the ACM, 64(12): 86-92.
  8. [17]17. Gilardi, F.; Alizadeh, M.; and Kubli, M. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Pro- ceedings of the National Academy of Sciences, 120(30): e2305016120.
  9. [18]18. Grimmer, J.; Roberts, M. E.; and Stewart, B. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press.
  10. [19]19. Grimmer, J.; and Stewart, B. M. 2013. Text as Data: The Promise and Pitfalls of Automatic Content Analysis Meth- ods for Political Texts. Political Analysis, 21(3): 267-297.
  11. [22]22. Huang, F.; Kwak, H.; and An, J. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv:2302.07736. · arXiv
  12. [23]23. Jacobs, A. Z.; and Wallach, H. 2021. Measurement and fair- ness. In Proceedings of the 2021 ACM conference on fair- ness, accountability, and transparency, 375-385.
  13. [26]26. Keith, K. A.; Teichmann, C.; O'Connor, B.; and Meij, E. 2020. Uncertainty over uncertainty: Investigating the as- sumptions, annotations, and text measurements of economic policy uncertainty. arXiv preprint arXiv:2010.04706. · arXiv
  14. [32]32. Li, M.; Shi, T.; Ziems, C.; Kan, M.-Y.; Chen, N. F.; Liu, Z.; and Yang, D. 2023. CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation. arXiv:2310.15638. · arXiv
  15. [35]35. McKinney, W. 2011. pandas: a foundational Python library for data analysis and statistics. Python for high performance and scientific computing, 14(9): 1-9.
  16. [39]39. Neuendorf, K. A. 2016. The Content Analysis Guidebook. Sage Publications.
  17. [40]40. Ollion, E.; Shen, R.; Macanovic, A.; and Chatelain, A. 2023. ChatGPT for Text Annotation? Mind the Hype! arXiv:https://osf.io/preprints/socarxiv/x58kn.
  18. [42]42. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; and et al., T. K. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems.
  19. [47]47. Shneiderman, B. 2020. Human-centered artificial intelli- gence: Reliable, safe and trustworthy. International Journal of Human-Computer Interaction, 6: 495-504.
  20. [49]49. Thapa, S.; Naseem, U.; and Nasim, M. 2023. From hu- mans to machines: can ChatGPT-like LLMs effectively re- place human annotators in NLP tasks. Workshop Proceed- ings of the 17th International AAAI Conference on Web and Social Media.
  21. [52]52. Wang, S.; Liu, Y.; Xu, Y.; Zhu, C.; and Zeng, M. 2021. Want to reduce labeling cost? GPT-3 can help. arXiv:2108.13487. · arXiv
  22. [54]54. Yang, K.-C.; and Menczer, F. 2023. Large language models can rate news outlet credibility. arXiv:2304.00228. · arXiv

Cited by

Sorted by citations

Related

Authors
Pangakis, N., & Wolken, S.
Affiliations
University of Pennsylvania (US)
Venue
Proceedings of the International AAAI Conference on Web and Social Media
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2025-06-07
Volume
19
Pages
1471-1492
ISSN
  • 2162-3449 (ISSN-L)
  • 2334-0770
DOI
10.1609/icwsm.v19i1.35883
OpenAlex
W4411120843
Cited by
27
References
20
Citations by year
2026: 20 · 2025: 6 · 2024: 1
FWCI
45.98
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the International AAAI Conference on Web and Social Media (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Princeton University
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
generative large language models, GPT-4, human-centered evaluation, social media research, computational social science, benchmark contamination, prompt tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review