Automated Annotation with Generative AI Requires Validation
Open accessAbstract
Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures, but their performance varies across annotation tasks due to prompt quality, text data idiosyncrasies, and conceptual difficulty. Because these challenges will persist even as LLM technology improves, we argue that any automated annotation process using an LLM must validate the LLM's performance against labels generated by humans. To this end, we outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. Using GPT-4, we validate this approach by replicating 27 annotation tasks across 11 datasets from recent social science articles in high-impact journals. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis. We make available easy-to-use software designed to implement our workflow and streamline the deployment of LLMs for automated annotation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Expert validation remains essential, especially as LLMs are increasingly integrated into research systems.
“Nonetheless, expert validation remains essential 40,41 , especially as LLMs are increasingly integrated into research systems 42-44 .”
From Quantifying the prevalence and impact of overreaching causal claims in social science (2026) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“As such, these tests are plausibly affected by contamination (Brown et al., 2020), meaning that the datasets may be included in the LLM’s training data and that strong performance may reflect memorization, which will not generalize to new datasets and tasks.”
BackgroundContent Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · see citing works
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 1× · see citing works
“…and expensive, human annotators, usually crowd-sourced workers 1 or undergraduate assistants, often suffer from a limited attention span, fatigue, and changing perceptions of the underlying conceptual categories throughout the labeling procedures (Grimmer and Stewart, 2013; Neuendorf, 2016).”
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 1× · see citing works
“…and expensive, human annotators, usually crowd-sourced workers 1 or undergraduate assistants, often suffer from a limited attention span, fatigue, and changing perceptions of the underlying conceptual categories throughout the labeling procedures (Grimmer and Stewart, 2013; Neuendorf, 2016).”
Codebook Development for Team-Based Qualitative Analysis
MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998) · CAM Journal · doi:10.1177/1525822x980100020301 · Cited by 1,675 · cited 1× · see citing works
“A rich literature spanning both qualitative and quantitative social science offers guidance on how to develop a codebook and classify text data based on relevant concepts (e.g., Krippendorff, 2018; Crabtree and Miller, 1992; MacQueen et al., 1998).”
BackgroundDouglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532 · cited 1× · see citing works
“…accuracy or intercoder agreement) and analyze 1 Researchers have raised concerns about the quality of data collected from workers on popular crowdsourcing platforms like Amazon’s Mechanical Turk (Chmielewski and Kucker, 2020; Douglas et al., 2023) a few insular tasks on a small number of datasets.”
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 1× · see citing works
“Gilardi et al. (2023) claim that ChatGPT outperforms MTurkers on a variety of annotation tasks.”
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · cited 2× · see citing works
“…accuracy or intercoder agreement) and analyze 1 Researchers have raised concerns about the quality of data collected from workers on popular crowdsourcing platforms like Amazon’s Mechanical Turk (Chmielewski and Kucker, 2020; Douglas et al., 2023) a few insular tasks on a small number of datasets.”
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · cited 2× · see citing works
“Numerous other studies present similar analyses with varying results and recommendations (e.g., He et al., 2023; Wang et al., 2021; Ziems et al., 2023; Zhu et al., 2023; Ding et al., 2022).”
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071873045 · Cited by 480 · see citing works
Want To Reduce Labeling Cost? GPT-3 Can Help
Wang, S., Liu, Y., Xu, Y., Zhu, C., & Zeng, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.354 · Cited by 208 · cited 1× · see citing works
“Numerous other studies present similar analyses with varying results and recommendations (e.g., He et al., 2023; Wang et al., 2021; Ziems et al., 2023; Zhu et al., 2023; Ding et al., 2022).”
ResultCan ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing TasksPreprint
Zhu, Y.-M., Zhang, P., Haq, E.-U., Hui, P., & Tyson, G. (2023) · arXiv.org · doi:10.48550/arxiv.2304.10145 · Cited by 183 · cited 2× · see citing works
“Numerous other studies present similar analyses with varying results and recommendations (e.g., He et al., 2023; Wang et al., 2021; Ziems et al., 2023; Zhu et al., 2023; Ding et al., 2022).”
BackgroundResultIs GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.626 · Cited by 149 · see citing works
Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary RemarkPreprint
Reiss, M. (2023) · doi:10.31219/osf.io/rvy5p · Cited by 149 · see citing works
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104 · cited 1× · see citing works
“Open-source LLMs represent a viable solution to many of these problems (Spirling, 2023).”
Card, D., Chang, S. Y., Becker, C., Mendelsohn, J., Voigt, R., Boustan, L. P., Abramitzky, R., & Jurafsky, D. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120510119 · Cited by 97 · cited 1× · see citing works
“As shown in Table 2, the aggregate performance ranged as low as an F1 2022), a difference of 0.552.”
InfluentialAnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Chen, L., Jiao, J., Yiu, S., Duan, N., & Chen, W. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-industry.15 · Cited by 97 · see citing works
Kennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., et al. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-021-09569-x · Cited by 90 · see citing works
On the rise of fear speech in online social media
Saha, P., Garimella, K. K., Kalyan, N. K., Pandey, S. K., Meher, P. M., Mathew, B., & Mukherjee, A. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2212270120 · Cited by 40 · see citing works
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2303.16854 · Cited by 34 · cited 1× · see citing works
“Numerous other studies present similar analyses with varying results and recommendations (e.g., He et al., 2023; Wang et al., 2021; Ziems et al., 2023; Zhu et al., 2023; Ding et al., 2022).”
ResultTesting the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
Reiss, M. V. (2023) · arXiv · doi:10.48550/arxiv.2304.11085 · Cited by 34 · cited 1×
“Reiss (2023), on the other hand, finds that LLMs perform poorly on text annotation and argues that this tool should be used cautiously.”
BackgroundPeople judge others to have more voluntary control over beliefs than they themselves do.
Cusimano, C. J., & Goodwin, G. P. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000198 · Cited by 30 · cited 2× · see citing works
“…cover an extensive range of social science applications, from identifying whether Cold War-era texts pertained to foreign affairs or military matters (Schub, 2022) to analyzing open-ended survey responses to classify how people conceptualize where beliefs come from (Cusimano and Goodwin, 2020).”
Informing the Leader: Bureaucracies and International Crises
Schub, R. (2022) · American Political Science Review · doi:10.1017/s0003055422000168 · Cited by 21 · cited 2× · see citing works
“…cover an extensive range of social science applications, from identifying whether Cold War-era texts pertained to foreign affairs or military matters (Schub, 2022) to analyzing open-ended survey responses to classify how people conceptualize where beliefs come from (Cusimano and Goodwin, 2020).”
BackgroundIs GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Joty, S., Li, B., & Bing, L. (2022) · arXiv · doi:10.48550/arxiv.2212.10450 · Cited by 15 · cited 1× · see citing works
“Numerous other studies present similar analyses with varying results and recommendations (e.g., He et al., 2023; Wang et al., 2021; Ziems et al., 2023; Zhu et al., 2023; Ding et al., 2022).”
ResultBalagopalan, A., Madras, D., Yang, D., Hadfield-Menell, D., Hadfield, G. K., & Ghassemi, M. (2023) · Science Advances · doi:10.1126/sciadv.abq0701 · Cited by 14 · cited 1× · see citing works
“For instance, whether a classification task is a factual question about a text sample or requires judgement has significant bearing on classification strategy (Balagopalan et al., 2023).”
Background
Flagged first, then by citations
Research works printed but not linked: 1
- Crabtree, B. F. and Miller, W. L. (1992). A template approach to text analysis: Developing and using codebooks. In Doing qualitative research. Sage Publications.
Cited by
Sorted by citations
Harnessing the power of AI in qualitative research: Exploring, using and redesigning ChatGPT
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100144 · Cited by 290 · see citing works
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 139 · see citing works
“Previous works have also studied the process of LLMs self-verifying their output, either by asking the models to self-evaluate their answers [35] or by measuring the uncertainty or confidence score associated with the answers [50, 69, 93, 101].”
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · see citing works
“…(110) discussing of LLM openness in detail as well as their openness ratings of current LLMs https://opening-up-chatgpt.github.io, including their specific openness criteria and evaluation strategies https://github.com/ opening-up-chatgpt/opening-up-chatgpt.github.io/tree/main/ projects#criteria.”
Hayes, A. S. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251322346 · Cited by 134 · see citing works
“Such mistakes underscore the importance of maintaining rigorous human oversight, cross-referencing the LLM ’ s outputs with original data, and ensuring that attributions are accurate to preserve the integrity of the analysis (Pangakis et al., 2023).”
Hate Speech Detection Using Large Language Models: A Comprehensive Review
Albladi, A., Islam, M., Das, A., Bigonah, M., Zhang, Z., Jamshidi, F., Rahgouy, M., Raychawdhary, N., Marghitu, D., & Seals, C. (2025) · IEEE Access · doi:10.1109/access.2025.3532397 · Cited by 72 · see citing works
“The researchers [106], [107] discussed community-specific LLMs for rule-based moderation, and validated LLMs for automated text annotation.”
Can LLMs Replace Manual Annotation of Software Engineering Artifacts?
Ahmed, T., Dévanbu, P., Treude, C., & Pradel, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00086 · Cited by 41 · see citing works
“LLMs for Data Synthesis and Assessment Beyond using LLMs to partially automate evaluations that would otherwise be performed by humans only, others propose to use LLMs to synthesize additional data to train or fine-tune models for text annotation tasks [64], [65].”
ChatGPT for Text Annotation? Mind the Hype!Preprint
Ollion, É., Shen, R., Macanovic, A., & Chatelain, A. (2023) · doi:10.31235/osf.io/x58kn · Cited by 33 · see citing works
Exploring the applicability of large language models to citation context analysis
Nishikawa, K., & Koshiba, H. (2024) · Scientometrics · doi:10.1007/s11192-024-05142-9 · Cited by 26 · see citing works
Validating the use of large language models for psychological text classification
Bunt, H. L., Goddard, A., Reader, T. W., & Gillespie, A. (2025) · Frontiers in Social Psychology · doi:10.3389/frsps.2025.1460277 · Cited by 19 · see citing works
“This is a problem because using LLMs for psychological text classification requires validation, as they can hallucinate and reproduce existing biases in the textual data they were trained on (Demszky et al., 2023; Pangakis et al., 2023).”
InfluentialMacanovic, A., & Przepiorka, W. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02381-9 · Cited by 17 · see citing works
“Still, on other tasks, the results of zero-shot classification with generative models need to be extensively validated (Kocoń et al., 2023; Ollion et al., 2023; Pangakis et al., 2023).”
BackgroundMethodHayes, A. S. (2023) · doi:10.31235/osf.io/yms8p · Cited by 12 · see citing works
Report on the Search Futures Workshop at ECIR 2024
Azzopardi, L. A., Clarke, C. L. A., Kantor, P. B., Mitra, B., Trippas, J. R., Ren, Z., Aliannejadi, M., Arabzadeh, N., Chandrasekar, R., de Rijke, M., et al. (2024) · ACM SIGIR Forum · doi:10.1145/3687273.3687288 · Cited by 11 · see citing works
“In addition, the reliability of these automated labels is still being examined [Ma et al., 2024; Pangakis et al., 2023].”
BackgroundSenator, F., Lakhfif, A., Zenbout, I., Boutouta, H., & Mediani, C. (2025) · IEEE Access · doi:10.1109/access.2025.3525493 · Cited by 15 · see citing works
“Research on the use of ChatGPT in English NLP is growing rapidly [8], [31], [32], supported by the availability of abundant resources and annotated corpora.”
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31219/osf.io/tg79n · Cited by 8 · see citing works
Automated vs. manual linguistic annotation for assessing pragmatic competence in English classes
Mahmoudi-Dehaki, M., & Nasr-Esfahani, N. (2025) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2025.100253 · Cited by 8 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31234/osf.io/d695y · Cited by 8 · see citing works
Stoll, A., Yu, J., Andrich, A., & Domahidi, E. (2025) · Communication Methods and Measures · doi:10.1080/19312458.2025.2551693 · Cited by 6 · see citing works
“Recent work has evaluated the validity of LLMs for automated content analysis (e.g., Laurer et al., 2024; Pangakis et al., 2023) and provided frameworks, tools, and application techniques for LLM-based methods (e.g., Gruber & Weber, 2024; Törnberg, 2024; Weber & Reichardt, 2023).”
Let’s talk about language—and its role for replicability
Schmalz, X., Breuer, J., Haim, M., Hildebrandt, A., Knöpfle, P., Leung, A. Y., & Roettger, T. B. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04381-2 · Cited by 7 · see citing works
Hong, E., & Yi, J.-S. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103673 · Cited by 6 · see citing works
Kwan, B. Y. M., Zhou, Z., Rogoza, N., Aghaei, N., de Vries, I., Hanmore, T., & Zevin, B. (2025) · Annals of Surgery Open · doi:10.1097/as9.0000000000000608 · Cited by 5 · see citing works
“27 Future studies should evaluate fine-tuned GPT-4.”
Beck, J., Kemeter, L. M., Dürrbeck, K., Abdalla, M. H. I., & Kreuter, F. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3528192 · Cited by 5 · see citing works
“Furthermore, the quality of the resulting annotations seems to be strongly data-and task-specific [49].”
Coding with the machines: machine-assisted coding of rare event data
Overos, H. D., Hlatky, R., Pathak, O., Goers, H., Gouws-Dewar, J., Smith, K., Chew, K. P., Birnir, J. K., & Liu, A. H. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae165 · Cited by 5 · see citing works
“Consequently, researchers call for LLM classifications to be validated (9, 12).”
BackgroundMethodInfluentialHaim, M., Hase, V., Schindler, J., Bachl, M., & Domahidi, E. (2023) · Studies in Communication and Media · doi:10.5771/2192-4007-2023-4-277 · Cited by 5 · see citing works
A Simple Ensemble Strategy for LLM Inference: Towards More Stable Text Classification
Niimi, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_17 · Cited by 8 · see citing works
LLM-enhanced Self-training for Cross-domain Constituency Parsing
Li, J., Zhang, M., Guo, P., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.508 · Cited by 4 · see citing works
Schepers, I., Bruijn, L. M., Wieling, M. B., & Vols, M. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09495-1 · Cited by 3 · see citing works
Government risk communication and response networks: Insights from typhoon news in China
Wang, L.-H., & Jiang, S.-Y. (2025) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2025.105587 · Cited by 3 · see citing works
Putting the AI into social science
Breuer, J. (2023) · AI critique · doi:10.1515/9783839467664-014 · Cited by 4 · see citing works
Andrade, G. H. B., Yada, S., & Aramaki, E. (2024) · JMIR Medical Informatics · doi:10.2196/59680 · Cited by 4 · see citing works
“Yet LLM annotation quality is still shy of what can be produced manually; thus, combining the automated technique and human effort is still the best approach to creating a high-quality data set [22].”
MethodEnhancing Text Annotation Through Rationale-Driven Collaborative Few-Shot Prompting
Wu, J., Wang, X., & Weijia Jia, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889661 · Cited by 7 · see citing works
“Traditional text annotation processes predominantly rely on manual labor, which is often time-consuming, labor-intensive, and prone to inconsistencies in data quality [2].”
Tripp, A. (2025) · Research & Politics · doi:10.1177/20531680251332353 · Cited by 2 · see citing works
Toward Automatic Dataset Discovery From Scientific Publications
Kumar, S., Ghosal, T., & Ekbal, A. (2025) · IEEE Access · doi:10.1109/access.2025.3532767 · Cited by 1 · see citing works
“These annotators often experience limited attention spans, fatigue, and shifting perceptions of the conceptual categories as labeling progresses [6], [7].”
Leveraging LLMs to Improve Human Annotation Efficiency with INCEpTION
Cunha, L. F., Yu, N., Silvano, P., Campos, R., & Jorge, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_10 · Cited by 1 · see citing works
A probabilistic model of hype reveals a rhetorical shift in biomedicine
Mishra, A., Diesner, J., & Torvik, V. I. (2026) · Quantitative Science Studies · doi:10.1162/qss.a.482 · Cited by 1 · see citing works
Engel, F., Giuliani, C., Watter, M., Kalantari, A., Schuller, K., Binder, H., & Kaier, K. (2026) · BMC Research Notes · doi:10.1186/s13104-026-07937-w · Cited by 1 · see citing works
“A shift towards more open annotation strategies appears viable, provided that free additions are scientifically validated [26].”
Mo, C., Manchaiah, V. K. C., Wasmann, J.-W. A., & Wang, S. (2026) · International Journal of Audiology · doi:10.1080/14992027.2026.2720841 · Cited by 1 · see citing works
Sudmann, A., Echterhölter, A., Ramsauer, M., Retkowski, F., Schröter, J., & Waibel, A. (2023) · AI critique · doi:10.1515/9783839467664 · Cited by 4 · see citing works
The quantum technology job market: data driven analysis of 3641 job posts
Goorney, S., Karydi, E., Muñoz, B., Santesson, O., Seskir, Z. C., Tudoran, A. A., & Sherson, J. (2026) · EPJ Quantum Technology · doi:10.1140/epjqt/s40507-026-00477-z · Cited by 1 · see citing works
Quantifying the prevalence and impact of overreaching causal claims in social science
Isch, C., Dörr, T., Fasching, N., Jennings, G., & Watts, D. J. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02553-x
Self-citationMacanovic, A., & Przepiorka, W. (2022) · doi:10.31235/osf.io/cerzs
ПРОГРАМУВАННЯ ЗА ДОПОМОГОЮ СИСТЕМ ГЕНЕРАТИВНОГО ШТУЧНОГО ІНТЕЛЕКТУ: РИЗИКИ ТА ВИКЛИКИ
БОЙКО, В., Василенко, М. Д., & Слатвінська, В. М. (2023) · INFORMATION TECHNOLOGY AND SOCIETY · doi:10.32689/maup.it.2023.2.2
Crowdsourced Homophily Ties Based Graph Annotation Via Large Language Model
Bu, Y., Zhu, Y., & Zhou, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889118
Automated Reddit Data Annotation with Large Language Models
Vuruma, S. K. R., Wu, D., Gupta, S. S., Aust, L., Lookingbill, V., Bellamy, W., Ren, Y., Kasson, E. M., Chen, L., Cavazos‐Rehg, P. A., et al. (2025) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi64645.2025.00049
“These LLMs have shown proficiency in in-context learning where they outperformed traditional methods [21], [22] and can generate quick results while not being susceptible to some of the limitations observed in human annotation [23].”
Reich, A., Thoms, C., & Schrimpf, T. (2025) · Communication Methods and Measures · doi:10.1080/19312458.2026.2693637 · Cited by 1 · see citing works
“Data coded by experts (Pangakis et al., 2023) or determined by majority decision from multiple coders (Törnberg, 2024b) have been recommended and tend to be of higher quality.”
InfluentialTowards Automatic ACT-Based Analysis of Chats
Bieberstein, A., Gampe, S., Schnitzer, B. L., & Korn, O. (2026) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3806774.3827978
Zhang, L., Wan, Z., Cheng, M., Shi, S., & Gou, Z. (2026) · Scientometrics · doi:10.1007/s11192-026-05813-9
“Some scholars argue that LLMs’ annotation logic aligns more closely with human reasoning, allowing them to outperform advanced supervised learning models in capturing context-based semantics (He et al., 2024; Pangakis et al., 2023; Reiss, 2023).”
Watter, M., Engel, F., Kalantari, A., et al. (2026) · Bioinformatics Advances · doi:10.1093/bioadv/vbag283
“%) highlights a persistent challenge where models may overlook entities in documents with high information density, reinforcing the necessity for expert post-validation to capture missing entries (Pangakis et al. 2023, Wang et al. 2024, Brokman et al. 2025).”
- Authors
- Pangakis, N., Wolken, S., & Fasching, N.
- Affiliations
- University of Pennsylvania (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.00176
- OpenAlex
- W4379255800
- Semantic Scholar
- e58231569f45704a2460bb6de3ec6f52efc5cf95
- Cited by
- 47
- References
- 22
- Citations by year
- 2026: 8 · 2025: 21 · 2024: 6 · 2023: 11 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- generative large language models, human annotation comparison, prompt engineering, GPT-4, annotation workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
