Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning
Open accessAbstract
This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT-3.5 and GPT-4, though still lagging behind fine-tuned GPT-3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the domain of text annotations in general, both proprietary and open-source LLMs have shown remarkable effectiveness.
“In the domain of text annotations in general, both proprietary and open-source LLMs have shown remarkable effectiveness [1, 2, 13, 18, 25].”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
Self-citationCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsPreprint
Hu, Z., Wang, L., Lan, Y.-H., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., & Lee, R. K.-W. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2304.01933 · Cited by 547 · see citing works
Structured Pruning of Large Language Models
Wang, Z.-H., Wohlwend, J., & Lei, T. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.1910.04732 · Cited by 363
Digitization of healthcare sector: A study on privacy and security concerns
Paul, M., Maglaras, L. A., Ferrag, M., & Almomani, I. (2023) · ICT express · doi:10.1016/j.icte.2023.02.007 · Cited by 284 · see citing works
Automated Text Classification of News Articles: A Practical Guide
Barberá, P., Boydstun, A., Linn, S., McMahon, R., & Nagler, J. (2020) · Political Analysis · doi:10.1017/pan.2020.8 · Cited by 156 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Inseq: An Interpretability Toolkit for Sequence Generation Models
Sarti, G., Feldhus, N., Sickert, L., van der Wal, O., Nissim, M., & Bisazza, A. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-demo.40 · Cited by 110 · see citing works
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104 · see citing works
The dangers of using proprietary LLMs for research
Ollion, É., Shen, R., Macanovic, A., & Chatelain, A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00783-6 · Cited by 62 · see citing works
Annotated dataset creation through large language models for non-english medical NLP
Frei, J., & Kramer, F. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104478 · Cited by 58 · see citing works
Content Moderation As a Political Issue: The Twitter Discourse Around Trump's Ban
Alizadeh, M., Gilardi, F., Hoes, E., Klüser, K. J., Kubli, M., & Marchal, N. (2022) · Journal of Quantitative Description Digital Media · doi:10.51685/jqd.2022.023 · Cited by 35 · see citing works
Self-citationMarchal, N., Hoes, E., Klüser, K. J., Hamborg, F., Alizadeh, M., Kubli, M., & Katzenbach, C. (2024) · Political Communication · doi:10.1080/10584609.2024.2377992 · Cited by 21 · see citing works
Self-citationAlizadeh, M., Hoes, E., & Gilardi, F. (2023) · Scientific Reports · doi:10.1038/s41598-023-40716-2 · Cited by 17 · see citing works
Self-citationIs GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Joty, S., Li, B., & Bing, L. (2022) · arXiv · doi:10.48550/arxiv.2212.10450 · Cited by 15 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- Binz, Marcel and Eric Schulz. 2023. 'Turning large language models into cognitive models.' arXiv:2306.03917 . · arXiv
- Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al. 2020. 'Language models are few-shot learners.' Advances in neural information processing systems 33:1877-1901.
- Card, Dallas, Amber Boydstun, Justin H Gross, Philip Resnik and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) . pp. 438-444.
- Chung, Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma et al. 2022. 'Scaling instruction-finetuned language models.' arXiv:2210.11416 . · arXiv
- Dettmers, Tim, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer. 2024. 'Qlora: Efficient finetuning of quantized llms.' Advances in Neural Information Processing Systems 36.
- He, Junxian, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick and Graham Neubig. 2021. 'Towards a unified view of parameter-efficient transfer learning.' arXiv:2110.04366 . · arXiv
- Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark et al. 2022. 'Training compute-optimal large language models.' arXiv:2203.15556 . · arXiv
- Howard, Jeremy and Sebastian Ruder. 2018. 'Universal language model fine-tuning for text classification.' arXiv:1801.06146 . · arXiv
- Hu, Edward J, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2021. 'Lora: Low-rank adaptation of large language models.' arXiv:2106.09685 . · arXiv
- Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa. 2022. 'Large language models are zero-shot reasoners.' arXiv:2205.11916 . · arXiv
- Murdoch, Blake. 2021. 'Privacy and artificial intelligence: challenges for protecting health information in a new era.' BMC Medical Ethics 22(1):1-5.
- Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems , ed. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho and A. Oh. Vol. 35 Curran Associates, Inc. pp. 27730-27744.
- Pangakis, Nicholas, Samuel Wolken and Neil Fasching. 2023. 'Automated Annotation with Generative AI Requires Validation.' arXiv:2306.00176 . · arXiv
- Ray, Partha Pratim. 2023. 'ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.' Internet of Things and Cyber-Physical Systems .
- Schick, Timo, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda and Thomas Scialom. 2024. 'Toolformer: Language models can teach themselves to use tools.' Advances in Neural Information Processing Systems 36.
- Shen, Yongliang, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu and Yueting Zhuang. 2024. 'Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.' Advances in Neural Information Processing Systems 36.
- Törnberg, Petter. 2023 b . 'Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.' arXiv:2304.06588 . · arXiv
- Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale et al. 2023. 'Llama 2: Open foundation and fine-tuned chat models.' arXiv:2307.09288 . · arXiv
- Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le and Denny Zhou. 2022. 'Chain of thought prompting elicits reasoning in large language models.' arXiv:2201.11903 . · arXiv
- Yang, Jingfeng, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin and Xia Hu. 2023. 'Harnessing the power of llms in practice: A survey on chatgpt and beyond.' arXiv:2304.13712 . · arXiv
- Yang, Wen, Chong Li, Jiajun Zhang and Chengqing Zong. 2023. 'BigTrans: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages.' arXiv:2305.18098 . · arXiv
- Zhang, Biao, Zhongtao Liu, Colin Cherry and Orhan Firat. 2024. 'When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method.' arXiv:2402.17193 . · arXiv
- Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis and Alex Smola. 2023. 'Multimodal chain-of-thought reasoning in language models.' arXiv:2302.00923 . · arXiv
Cited by
Sorted by citations
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
If in a Crowdsourced Data Annotation Pipeline, a GPT-4
He, Z.-Y., Chieh-Yang, H., Ding, C. C., Rohatgi, S., & Huang, T. (2024) · International Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642834 · Cited by 81 · see citing works
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63 · see citing works
“Fine-tuning can potentially enhance the performance of LLMs by adapting them more speci fi cally to the nuances of the task and data at hand (Alizadeh et al. 2024).”
The shrinking landscape of linguistic diversity in the age of large language models
Sourati, Z., Karimi-Malekabadi, F., Ozcan, M., McDaniel, C., Ziabari, A. S., Trager, J., Tak, A. N., Chen, M.-H., Morstatter, F., & Dehghani, M. (2025) · Nature Human Behaviour · doi:10.1038/s41562-026-02550-0 · Cited by 47
“We used OpenAI’s text-embedding-ada-002 model to generate high-dimensional embeddings for both the original and rewritten texts because of their excellent performance on sentence similarity tasks 113,114 and near-human annotation capabilities 115,116 .”
Large Language Models for Propaganda Span Annotation
Hasanain, M., Ahmed, F., & Alam, F. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2311.09812 · Cited by 43
“Alizadeh et al. (2023) 2023) explored ChatGPT’s ability to detect implicitly hateful tweets and to generate natural language explanations for this task.”
BackgroundSimple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task ExecutionPreprint
Alizadeh, M., Samei, Z., Stetsenko, D., & Gilardi, F. (2025) · arXiv.org · doi:10.48550/arxiv.2506.01055 · Cited by 39 · see citing works
“2 Related work LLM-integrated Systems LLMs have achieved impressive results in tasks such as question answering [44], machine translation [51], text annotation [15, 3], and summarization [49], drawing widespread interest from both academia and industry.”
Self-citationKarim, M. M., Khan, S., Van, D. H., Liu, X.-Y., Wang, C.-H., & Qu, Q. (2025) · Future Internet · doi:10.3390/fi17080353 · Cited by 37 · see citing works
“Illustrative studies include [53], who evaluated LLMs against human annotators in relevance detection using precision metrics and Cohen’s kappa to quantify agreement levels.”
A Comprehensive Survey on LLM‐Based Network Management and Operations
Hong, J., Tu, N. V., & Hong, J. W. (2025) · International Journal of Network Management · doi:10.1002/nem.70029 · Cited by 27 · see citing works
Sapkota, R., Raza, S., & Karkee, M. (2025) · arXiv.org · doi:10.48550/arxiv.2502.18505 · Cited by 25 · see citing works
“…presenting the current scenarios and future paths as illustrated in Figure 6 The classification of LLMs as open weights rather than open source is primarily driven by the selective disclosure of components in the model development process Liesenfeld and Dinge-manse (2024); Alizadeh et al. (2025).”
SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models
Peri, R., Jayanthi, S. M., Ronanki, S., Bhatia, A., Mundnich, K., Dingliwal, S., Das, N., Hou, Z., Huybrechts, G., Vishnubhotla, S., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2405.08317 · Cited by 25
“This method enables us to effectively compare different SLMs as well as employ it for identifying jailbroken instances at par with crowd-sourcing (Alizadeh et al., 2023; Shen et al., 2024; Tan et al., 2024).”
MethodWu, J., Fan, X., Lu, B.-R., Jiang, X., Mesgarani, N., Hasegawa‐Johnson, M., & Mari Ostendorf, E. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832300 · Cited by 3 · see citing works
“This is because either (1) only one preference/event/action is provided and associated to a single speaker in the conversation, or (2) the preference/event/action of the two speakers are the same.”
Adversarial Text Purification: A Large Language Model Approach for Defense
Moraffah, R., Khandelwal, S., Bhattacharjee, A., & Liu, H. (2024) · Pacific-Asia Conference on Knowledge Discovery and Data Mining · doi:10.48550/arxiv.2402.06655 · Cited by 18
“LLMs as pseudo-oracles: Alongside the impressive performance of LLMs on a variety of natural language tasks [7], LLMs are also being increasingly used as pseudo-oracles, such as in data annotation [11,1,14], as detectors [4], for model explainability [5] and as experts in general [34].”
MethodCan LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs
Zhou, W., Zhou, J., Wang, H., Li, Z., He, Q.-K., Han, S.-K., Li, G.-L., Zhou, X.-H., He, Y.-Y., Liu, C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.17058 · Cited by 17
“OpenLLMAnno [83] demonstrates that fine-tuned local LLM S (e.g., Llama 2 , FLAN-T5 ) outperform proprietary models like GPT-3.”
Nasution, A., Onan, A., Murakami, Y., Monika, W., & Hanafiah, A. (2025) · IEEE Access · doi:10.1109/access.2025.3574629 · Cited by 17 · see citing works
“Alizadeh et al. provided a practical guide for using open-source LLMs in text annotation, discussing optimal settings and fine-tuning strategies for improved accuracy and efficiency [28].”
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16 · see citing works
“In the domain of text annotations in general, both proprietary and open-source LLMs have shown remarkable effectiveness [1, 2, 13, 18, 25].”
BackgroundModelling Political Coalition Negotiations Using LLM-based AgentsPreprint
Moghimifar, F., Li, Y.-F., Thomson, R., & Haffari, G. (2024) · arXiv.org · doi:10.48550/arxiv.2402.11712 · Cited by 16 · see citing works
“With recent advances in LLMs, there’s been a surge in using such models to replace manual processes in political science (Alizadeh et al., 2023).”
MethodGhorbian, M., Ghobaei-Arani, M., & Ghorbian, S. (2025) · Methods · doi:10.1016/j.ymeth.2025.04.001 · Cited by 14 · see citing works
Fung, M. M. H., Tang, E. H. M., Wu, T., Luk, Y., Au, I., Liu, X.-D., Lee, V. H. F., Wong, C.-K., Wei, Z., Cheng, W., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01528-y · Cited by 13 · see citing works
Automatic feedback on physics tasks using open-source generative artificial intelligence
Meyer, A., Bleckmann, T., & Friege, G. (2025) · International Journal of Science Education · doi:10.1080/09500693.2025.2499220 · Cited by 12 · see citing works
“Developers can use fine tuning or similar techniques independently to significantly improve the LLM’s performance for specific tasks (e.g. Alizadeh et al., 2025).”
Li, Y., Tsang, Y.-P., Ho, D. C., Ozden, M., Lee, C. K. H., & Hu, H. (2025) · Enterprise Information Systems · doi:10.1080/17517575.2025.2462069 · Cited by 12 · see citing works
A GPT among Annotators: LLM-based Entity-Level Sentiment Annotation
Rønningstad, E., Velldal, E., & Øvrelid, L. (2024) · Law · doi:10.18653/v1/2024.law-1.13 · Cited by 11
Gonz'alez-Bustamante, B. (2024) · arXiv.org · doi:10.48550/arxiv.2409.09741 · Cited by 10 · see citing works
“5 and 4 in zero-shot tasks, only lagging in fine-tuned GPTs (Alizadeh et al., 2024).”
BackgroundGlobal-Liar: Factuality of LLMs over Time and Geographic Regions
Mirza, M. S., Coelho, B., Cui, Y.-Y., Pöpper, C., & McCoy, D. (2024) · arXiv.org · doi:10.48550/arxiv.2401.17839 · Cited by 10
Zhuang, Y., Wang, F.-L., Chiu, D. K. W., & Ho, K. K. W. (2025) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2025.110602 · Cited by 9 · see citing works
Chang, C. H., Cook-Patton, S., Erbaugh, J., Lu, L., Masuda, Y. J., Molnár, I., Papp, D., & Robinson, B. E. (2025) · Conservation Biology · doi:10.1111/cobi.14464 · Cited by 9 · see citing works
“Therefore, Chang et al. (2025) were uncertain which pathway and impact combinations would even exist as valid categories.”
InfluentialA validity-guided workflow for robust large language model research in psychology
Lin, Z. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03073-2 · Cited by 8 · see citing works
“Based on the research goal defined in Stage 1, the validation process diverges into two pathways.”
Gender Dynamics and Homophily in a Social Network of LLM Agents
Fadaei, F., Moran, J., & Yasseri, T. (2026) · arXiv.org · doi:10.1098/rsta.2025.0205 · Cited by 8
“Recent work shows that large language models can serve as high-quality automated annot-ators and, in many cases, match or outperform human crowd workers on text annotation and evaluation tasks [1, 13, 21, 34].”
Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences
Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00843-4 · Cited by 13 · see citing works
“Researchers in the area of text annotation provide specific suggestions for prompt development in that case 54,55 ; we suggest that researchers follow their lead, establishing similar types of suggestion for their particular use cases.”
InfluentialLAMeD: LLM-generated Annotations for Memory Leak Detection
Shemetova, E., Shenbin, I., Smirnov, I., Alekseev, A. M., Rukhovich, A. D., Nikolenko, S. I., Lomshakov, V., & Piontkovskaya, I. (2025) · International Conference on Evaluation & Assessment in Software Engineering · doi:10.1145/3756681.3756999 · Cited by 7 · see citing works
Alkaeed, M., Abioye, S., Qayyum, A., Mekki, Y., Berrou, I., Abdallah, M. M., Al-Fuqaha, A. I., Bilal, M., & Qadir, J. (2025) · arXiv.org · doi:10.48550/arxiv.2502.04356 · Cited by 7
“In the ML research and development community, doubts remain about the merits and drawbacks of making LLMs and AIFMs open-source or retaining them as proprietary models [25], [33].”
Validity and Effectiveness of E-Module-Based Learning Media in Informatics Subjects
Oksila, D., Giatman, Irfan, D., & Alawyah, K. (2025) · Jurnal Penelitian Pendidikan IPA · doi:10.29303/jppipa.v11i2.9861 · Cited by 6
“The practicality of learning media is used to measure the extent of the practicality of the learning media (Dai, 2025; Alizadeh et al., 2025).”
Ensemble of Large Language Models for Curated Labeling and Rating of Free-text Data
Qiu, J., Guo, D., Natalie, P., Noelle, P., Levinson, C., & Henry, T. R. (2025) · arXiv.org · doi:10.48550/arxiv.2501.08413 · Cited by 5
“Alizadeh et al. [14] demonstrated that fine-tuning can enhance the performance of open-source LLMs in text classification tasks within political science, getting closer to the performance of zero-shot prompting with GPT-3.”
Meng, X., Wang, X.-H., & Peng, T.-Q. (2026) · Communication Methods and Measures · doi:10.1080/19312458.2026.2613829 · Cited by 5 · see citing works
“…models are designed to learn associations between predefined features and target categories, LLM-based annotation is more akin to human coding, in that labels are generated based on an understanding of the semantic and pragmatic context (Alizadeh et al., 2024; Törnberg, 2024; Wei et al., 2023).”
Uncovering the Trust Signals Supporting Telegram’s Cybercrime Economy
Ricaldi, R., Marjanov, T., Allodi, L., & Hutchings, A. (2025) · APWG Symposium on Electronic Crime Research · doi:10.1109/ecrime66972.2025.11327769 · Cited by 5
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
Tavakoli, L., & Zamani, H. (2025) · International Conference on the Theory of Information Retrieval · doi:10.1145/3731120.3744574 · Cited by 5 · see citing works
A Case Study on the Generative AI Project Life Cycle Using Large Language Models
Bandi, A., & Kagitha, H. (2024) · EPiC series in computing · doi:10.29007/hvzc · Cited by 10 · see citing works
“Projects like LLAMA offer readily accessible resources that can be modified and deployed within our environments [1].”
BackgroundLLMs Can Patch Up Missing Relevance Judgments in Evaluation
Upadhyay, S., Kamalloo, E., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2405.04727 · Cited by 4
“Even, the open-source LLMs have provided results comparable with proprietary models in the task of text annotations [1].”
MethodGhorbian, M., & Ghobaei-Arani, M. (2025) · Journal of Big Data · doi:10.1186/s40537-025-01304-5 · Cited by 4 · see citing works
A Hybrid Approach to Conangtent Generation Based on User Experience Using Generative AI Elements
Shayegan, M., & Hosseinpour, H. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100722 · Cited by 4 · see citing works
Zhong, C., Ye, F., Wang, Z., Jigeer, A., & Zhan, Z. (2025) · International Conference on Educational and Information Technology · doi:10.1109/iceit64364.2025.10975892 · Cited by 4 · see citing works
“Secondly, LLMs can generate text that aligns with specific patterns based on task requirements and contextual information [12], establishing a foundation for the automated generation of high-quality interdisciplinary test items.”
Deep literature reviews: an application of fine-tuned language models to migration research
Iacus, S., Qi, H., & Han, J. (2025) · arXiv.org · doi:10.48550/arxiv.2504.13685 · Cited by 3
“This technique not only reduces computational costs but also enhances the model’s relevance for specialized tasks [10].”
Secret Leak Detection in Software Issue Reports using LLMs: A Comprehensive Evaluation
Ahmed, S., Rahman, M. N., Wahab, Z., Uddin, G., & Shahriyar, R. (2024) · IEEE Working Conference on Mining Software Repositories · doi:10.1145/3793302.3793348 · Cited by 3
Ma, C.-X., Peng, F.-L., Li, Z.-J., Zhang, T.-Z., & Qiao, Y. (2026) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2025.107390 · Cited by 3 · see citing works
Liu, F., Lai, T. Y. E., Xu, W.-X., & Li, G.-D. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349393 · Cited by 2 · see citing works
“To rigorously evaluate the performance of our proposed ViralMultiNet, we compared it against several advanced baseline models encompassing both unimodal and multimodal architectures [31,8].”
Yu, S.-J., Lyu, M., Ma, H., Xu, K.-Y., Li, H.-Y., Ma, L.-L., & Ji, M.-T. (2026) · Journal of Medical Internet Research · doi:10.2196/88758 · Cited by 2 · see citing works
“Recent methodological advancements, such as collaborative digital frameworks [31] and the use of LLMs in text interpretation [32-35], have sought to enhance analytic rigor and transparency.”
Public perspectives on food date labeling: implications for waste reduction and food safety policy
Lu, P., Mao, L.-L., Zhuang, X.-T., Xu, Z.-H., & Ma, S. (2026) · Food Policy · doi:10.1016/j.foodpol.2026.103055 · Cited by 2 · see citing works
An Integrated AI Framework for Crop Recommendation
Youssef, S., Gamage, K., & Zablith, F. (2026) · Horticulturae · doi:10.3390/horticulturae12040416 · Cited by 2
Improving hate speech detection with large language models
Umansky, N., Kubli, M., Kotarcic, A., Bronner, L., Kurer, S., Grech, P., Hangartner, D., Gilardi, F., & Donnay, K. (2026) · European Journal of Political Research · doi:10.1017/s1475676525100546 · Cited by 2 · see citing works
Self-citationEr, D., Yayla, K., & Aktas, B. (2026) · Nursing Inquiry · doi:10.1111/nin.70078 · Cited by 2 · see citing works
Efficient Text Classification with Conformal In-Context Learning
Pantelidis, I., Randl, K., & Henriksson, A. (2025) · arXiv.org · doi:10.48550/arxiv.2512.05732 · Cited by 2
“…settings, several studies show that fine-tuning smaller models still achieves superior performance on many text classification tasks, suggesting that LLMs are not yet universally state-of-the-art for classification (Edwards & Camacho-Collados, 2024; Bucher & Martini, 2024; Alizadeh et al., 2023).”
Automatic and Semiautomatic Methods for Domain Knowledge-Graph Construction and Ontology Expansion
Khalov, A., & Ataeva, O. (2025) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105525701547 · Cited by 2
Using text to analyze team dynamics: revisiting McGrath’s ABCs
Kush, J., Kane, A. A., Lehmann-Willenbrock, N., Paletz, S. B. F., & van Swol, L. V. (2025) · Team Performance Management · doi:10.1108/tpm-02-2025-0020 · Cited by 2 · see citing works
Towards scalable and reliable coding of semantic property norms: ChatGPT vs. an improved AC-PLT
Ramos, D., Moreno, S., Canessa, E., & Chaigneau, S. E. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02838-5 · Cited by 2
“…to the growing popularity and strong appeal of artificial intelligence (AI) and large language models (LLMs) for solving tasks fairly related to our coding problem (Alizadeh et al. 2025; Liyanage, Gokani, & Mago, 2024), we compared the performance of our improved AC-PLT with that of the LLM gpt-3.”
LaajMeter: A Framework for LaaJ Evaluation
Ackerman, S., Amram, G., Fandina, O., Farchi, E., Froimovich, S., Gal, R., Ibraheem, W., & Ziv, A. (2025) · arXiv.org · doi:10.48550/arxiv.2508.10161 · Cited by 2
“The work [2] explores the use of open-source LLMs for text annotation in political science, highlighting the effectiveness of fine-tuning in low-resource settings.”
Megahed, F., Chen, Y.-J., Jones-Farmer, L. A., Lee, Y., Wang, J.-W., & Zwetsloot, I. (2025) · The Journal of Business Analysis · doi:10.48550/arxiv.2505.14918 · Cited by 2
“The framework provides guidance on sample size planning, prompt design, model selection criteria, and a methodology to assess reliability and validity [2].”
SEEBot: Leveraging Open-Source LLMs and RAG for Secure and Economical Enterprise Chatbots
Hamayat, F., Ejaz, L., Danish, M., Nazir, A., Ahadian, P., & Ahmad, R. F. (2025) · 2025 5th International Conference on Artificial Intelligence and Education (ICAIE) · doi:10.1109/icaie64856.2025.11158323 · Cited by 2
“Today, LLMs power many AI-driven solutions, including ChatGPT, making AI more accessible and interactive for users across different domains [5], [6].”
Synthetic News Generation for Fake News Classification
Sittar, A., Golob, L., & Smiljanic, M. (2025) · doi:10.48550/arxiv.2503.24206 · Cited by 2
“Moreover, controlled dataset creation ensures a more balanced and systematic training set, improving model robustness and generalization in real-world detection tasks [10], [11].”
Popov, D., Terentev, E., Serenko, D., Sochenkov, I. V., & Buyanov, I. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc9050116 · Cited by 2 · see citing works
“Regularly, researchers issue the best practices about how to obtain the best from LLMs as annotators, like in [21].”
Entity-Level Sentiment: More than the Sum of Its Parts
Rønningstad, E., Klinger, R., Velldal, E., & Øvrelid, L. (2024) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.48550/arxiv.2407.03916 · Cited by 2
“Recent work indicates that ChatGPT and open-source counterparts may be a relevant resource for annotating and labeling English texts (Gilardi et al., 2023; Alizadeh et al., 2024).”
BackgroundRevisiting business associations in trade politics: a large language model scaling approach
Cezar, R. F. (2026) · Review of International Political Economy · doi:10.1080/09692290.2026.2706545 · Cited by 1 · see citing works
Ma, C.-X., Peng, F.-L., Li, Z.-J., & Qiao, Y. (2026) · ENGINEERING Structure and Civil Engineering · doi:10.1007/s11709-026-1329-1 · Cited by 1 · see citing works
How backsliding government parties defend democratic backsliding in the European Parliament
Winzen, T. (2026) · European Union Politics · doi:10.1177/14651165261463883 · Cited by 1
Alizadeh, M., Gilardi, F., Mosleh, M., & Kasneci, E. (2026) · arXiv.org · doi:10.48550/arxiv.2606.11456 · Cited by 1
“Large language models have proven effective for a range of social science tasks, including text classification [31], content coding [32, 33], survey-response simulation [30], and qualitative-data analysis [34, 35].”
Self-citationExploring the architectures of large language models and impact across multiple domains: a review
Waqas, M., Rajab, R., Laghari, A. A., Kryvinska, N., Hejaili, A. A., Khan, N., Khuhro, M., & Zhang, F.-L. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11609-3 · Cited by 1 · see citing works
“Open-source models like Meta’s LLaMA, EleutherAI’s GPT-Neo, and Mistral give the opportunity to researchers to build their own applications without leaning on big compa-1 nies’ systems (Alizadeh et al. 2025; Habibi 2025).”
Large language models struggle with ethnographic text annotation
Goodall, L. S., Shilton, D. O., Mullins, D., & Whitehouse, H. (2026) · arXiv.org · doi:10.48550/arxiv.2601.12099 · Cited by 1
“Given limited labelled data and to prioritise usability for non-technical researchers, we adopt prompting-based strategies rather than fine-tuning the models’ internal parameters, despite the latter’s known performance benefits [68].”
Reich, A., Thoms, C., & Schrimpf, T. (2025) · Communication Methods and Measures · doi:10.1080/19312458.2026.2693637 · Cited by 1 · see citing works
“After all, there are many factors involved in their performance so that differences between models cannot be automatically ruled out (Alizadeh et al., 2024).”
Wass, L. M., Wu, Z., Vizoso, J., Wu, J. T. K., & Lin, L. K. (2026) · Journal of Medical Internet Research · doi:10.2196/80346 · Cited by 3 · see citing works
“The benchmark size was selected to support comparative evaluation of model performance under realistic annotation constraints, in line with previous evaluations of LLMs for short text annotations [42-44].”
Sekwenz, M.-T., Gsenger, R., Stocker, V., Görnemann, E., Talypova, D., Parkin, S., Greminger, L., & Smaragdakis, G. (2025) · Symposium on Human-Computer Interaction for Work · doi:10.1145/3707640.3731929 · Cited by 1 · see citing works
Hill, M., & Puschmann, C. (2025) · Publizistik · doi:10.1007/s11616-025-00891-w · Cited by 1 · see citing works
“LLMs are increasingly at the center of attention in media and communication research for their potential to advance content analysis (Alizadeh et al. 2024; Ziems et al. 2024).”
Human-LLM Collaborative Construction of a Cantonese Emotion Lexicon
Zhang, Y.-S., Dong, D., Hung, C.-T., Heyerdahl, L. W., Giles-Vernick, T., & Yeoh, E. (2024) · arXiv.org · doi:10.48550/arxiv.2410.11526 · Cited by 1
“Studies [8-10] have validated the cost-effectiveness and competitive potential of using LLMs for automatic annotation tasks.”
BackgroundLLMs for Targeted Sentiment in News Headlines: Exploring Different Levels of Prompt Prescriptiveness
Juros, J., Majer, L., & Snajder, J. (2024) · arXiv.org · doi:10.48550/arxiv.2403.00418 · Cited by 1
“The recent utilization of LLMs as data annotators (Wang et al., 2021; Pangakis et al., 2023; Alizadeh et al., 2023) invites a direct comparison between these paradigms and prompt design: a simple, less prescriptive prompt grants the LLM more freedom in interpreting the input, whereas a detailed,…”
MethodKeck, F., Broadbent, H., & Altermatt, F. (2026) · Ecology and Evolution · doi:10.1002/ece3.74626
“Small-scale LLMs—when appropriately fine-tuned— can achieve impressive results across different applications (Alizadeh et al. 2024; Carammia et al. 2024; Zhang et al. 2024).”
Cheimariotis, I., Costa, J. N., Piazza, F., Lodi, C., Rataj, M., Zawieska, J., Stępniak, M., & Grosso, M. (2026) · European Transport Research Review · doi:10.1186/s12544-026-00854-w
“…have matched or exceeded crowd-sourced annotators on relevance, stance, topic and frame detection and on political-science text classification [11, 12]; zero-shot, few-shot, fine-tuned and instruction-tuned approaches have been compared systematically [13, 14]; and data-efficient alternatives…”
Zamanifard, S., Goudarzi, S., & Robb, A. C. (2026) · doi:10.48550/arxiv.2610.27448
“Although labeled examples can help LLMs attend to task-relevant cues, their effectiveness is highly task-and context-dependent [1].”
Henn, J., Stoll, A., Feodorovici, P., Subramani, D., Röttgen, J., Arensmeyer, J. C., Gräff, I., Wulff, B., Kalff, J. C., & Matthaei, H. (2026) · Scientific Reports · doi:10.1038/s41598-026-73738-7
Chandra, Y., & Tan, J. (2026) · Policy Studies Journal · doi:10.1111/psj.70165
Carammia, M., Iacus, S., & Porro, G. (2026) · Social science computer review · doi:10.1177/08944393261485320
Palicki, S.-K., Bravo, I., & Claessen, C. (2026) · Social Science Computer Review · doi:10.1177/08944393261473230
de Cooker, J. (2026) · Journalism and Media · doi:10.3390/journalmedia7030173
Molas, C. F. (2026) · European Union Politics · doi:10.1177/14651165261470222
Gong, Y., Goyal, A., Ravikumar, S., & Wang, D. (2026) · Int. J. Hum. Comput. Stud. · doi:10.1016/j.ijhcs.2026.103837
Turan, S. C., Çıplak, Z., Sarıkaş, A., & Yıldız, K. (2026) · Journal of Engineering Research · doi:10.1016/j.jer.2026.08.006
Marino, G., Paroni, B., & Giglietto, F. (2026) · Information, Communication & Society · doi:10.1080/1369118x.2026.2696929
Benchmarking large language models for target-based financial sentiment and stock return
Muhammad, I., Rospocher, M., Knez, T., & Žitnik, S. (2026) · International Journal of Data Science and Analysis · doi:10.1007/s41060-026-01201-x
“Furthermore, recent research highlights the efficacy of smaller, computationally efficient models, attaining performance akin to larger LLMs via focused fine-tuning [33, 34].”
Andreas, L., Cokro, R. M., Gunawan, A. A. S., & Pradana, R. (2026) · 2026 5th International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666738
“Alizadeh et al. [5] provided practical guidance on open-source LLM annotation covering model selection and prompt design considerations central to this study's experimental design.”
Türkiye'de Alevilerin Medyada Temsili: Haber Tweetleri Üzerine Hesaplamalı Bir Analiz
Demirel, S. (2026) · Yeni Medya Dergisi · doi:10.55609/yenimedya.1865257
Adam, R., Maier, D., & Kogler, M. L. (2026) · arXiv.org · doi:10.48550/arxiv.2606.26489
“Alizadeh et al. (2025) 2025) framed context engineering as a broader discipline encompassing retrieval, processing, and management of all contextual information provided to LLMs during inference.”
Automatic Transformation of Textual Content Using LLM-Technology: a Feasibility Study
Smit, K., Leewis, S., van de Hoef, A., & Castro, D. (2026) · Bled eConference · doi:10.18690/um.fov.4.2026.35
Exploring migration research with large language model: methods and insights
Iacus, S., Qi, H.-D., & Han, J. (2026) · Royal Society Open Science · doi:10.1098/rsos.251577
Framing Migration News with LLMs: Structured CoT as a Support for Human Interpretation
del Barrio, D. A., Wen, J., & Gática-Pérez, D. (2026) · The Compass · doi:10.1145/3811242.3819110
annotaid: A browser-based tool for LLM-assisted qualitative coding of open text
Sprengholz, P. (2026) · SoftwareX · doi:10.1016/j.softx.2026.102702
Wu, N., Yang, B., & Zhang, D.-P. (2026) · Journal of planning education and research · doi:10.1177/0739456x261442161
The extended community of diet culture around eating disorders: a network analysis of reddit forums
Qiu, J., Levinson, C., Kropko, J., Henry, T. R., & Gates, A. J. (2026) · Applied Network Science · doi:10.1007/s41109-026-00787-2
“…language models, which provide a more direct and reliable way to detect the presence of our predefined topics in social media text (Chang et al. 2024; Alizadeh et al. 2024; Tai et al. 2024; Tan et al. 2024), compared to traditional topic modeling techniques in such settings (Laureate et al. 2023).”
Mekhzoumi, D., Mehidi, S., Amroun, K., Boulkaboul, S., & Medjebar, T. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21511-3
Zhu, G., Trofimova, G., Boguslavskaya, V., & Yu, H. (2026) · 2026 Communication Strategies in Digital Society Seminar (ComSDS) · doi:10.1109/comsds69583.2026.11493808
“In terms of practical application, Alizadeh [6] provides guidelines for using open-source models for text annotation, while Deiner [7] demonstrates the potential of LLMs in "Inductive Thematic Analysis," proving that models can not only perform deductive classification but also assist in…”
Evaluating AI models for food and alcohol advertisement classification against human benchmarks
Gitu, P.-A., Cerina, R., Grigoriev, A., & Vandevijvere, S. (2026) · Scientific Reports · doi:10.1038/s41598-026-42426-x
“For example, with careful configuration, open-source LLMs approached trained annotator performance and often surpassed crowd accuracy in a dual-benchmarking setting 11 .”
Sriveni, B., P, P., & Muppidi, S. (2026) · 2026 IEEE International Conference on Emerging Computing and Intelligent Technologies (ICoECIT) · doi:10.1109/icoecit68303.2026.11497436
A Lightweight LLM-Based Semantic–Spatial Inference Framework for Fine-Grained Urban POI Analysis
Huang, Z., Guo, Y.-X., Huang, S., & Zhao, M.-X. (2026) · Smart Cities · doi:10.3390/smartcities9010013
“Consequently, a lightweight, open, and scalable pathway re‑ mains lacking for expanding fine‑grained POI labels and converting semantic outputs into interpretable spatial evidence under realistic resource constraints [28,29].”
L’autonomie stratégique européenne entre mode discursive et besoin géopolitique
RĂDUŢ-GAGHI, L., & Tayebi, A. (2025) · Studia Universitatis Babeș-Bolyai Ephemerides · doi:10.24193/subbeph.2025.01
“». le consensus pour dépasser la « barrière de l’échelle » en recherche qualitative, en offrant une classification robuste à une fraction du coût du codage manuel (Alizadeh et al. 2024). récentes selon lesquelles les LLM « surpasseraient les codeurs experts » (Törnberg, 2025).”
- Authors
- Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.02179
- OpenAlex
- W4383473003
- Cited by
- 18
- References
- 17
- Citations by year
- 2025: 5 · 2024: 11 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- open-source large language models, text classification, textual annotation, fine-tuning, zero-shot learning, few-shot learning, stance detection, topic classification, relevance classification, political science research, news articles, tweets, GPT-3.5, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
