CoChat
Sign Up
Cited by 80Computational and Text Analysis MethodsTopic ModelingSentiment Analysis and Opinion Mining

Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning

Open access
Alizadeh, Meysam, Kubli, Maël, Samei, Zeynab, Dehghani, Shirin, Zahedivafa, Mohammadmasiha, Bermeo, Juan D., Korobeynikova, Maria, Gilardi, Fabrizio·2025·Journal of Computational Social Science
DOI 10.1007/s42001-024-00345-9PMC PMC11655591
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis and to establish a baseline performance benchmark that demonstrates the models’ effectiveness. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT $$-$$ - 3.5 and GPT-4, though still lagging behind fine-tuned GPT $$-$$ - 3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s42001-024-00345-9

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. [1]Alghisi, S., Rizzoli, M., Gabriel, R., Seyed MM., & Giuseppe R. (2024) Should we fine-tune or rag? evaluating different techniques to adapt llms for dialogue. arXiv preprint arXiv:2406.06399 · arXiv
  2. [2]Alizadeh, M., Gilardi, F., Emma H., K.üser, K.J., Kubli, M., & Marchal, N. (2022) Content moderation as a political issue: the twitter discourse around trump’s ban. Journal of Quantitative Description: Digital Media,2,
  3. [3]Alizadeh, Meysam, Hoes, Emma, & Gilardi, Fabrizio. (2023). Tokenization of social media engagements increases the sharing of false (and other) news but penalization moderates it. Scientific Reports, 13 (1), 13703. 37607955 PMC10444751 · DOI
  4. [4]Barberá, Pablo, Boydstun, Amber E., Linn, Suzanna, McMahon, Ryan, & Nagler, Jonathan. (2021). Automated text classification of news articles: a practical guide. Political Analysis, 29 (1), 19–42. · DOI
  5. [5]Binz, M., & Eric S. (2023) Turning large language models into cognitive models. arXiv preprint arXiv:2306.03917. · arXiv
  6. [6]Brown, Tom, Mann, Benjamin, Ryder, Nick, Subbiah, Melanie, Kaplan, Jared D., Dhariwal, Prafulla, Neelakantan, Arvind, Shyam, Pranav, Sastry, Girish, Askell, Amanda, et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877–1901.
  7. [7]Card, D., Boydstun, A., Gross, J.H., Resnik, P., & Smith, N.A. (2015). The media frames corpus: annotations of frames across issues. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), 438-444.
  8. [10]Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2024). Qlora: efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36.
  9. [12]Frei, Johann, & Kramer, Frank. (2023). Annotated dataset creation through large language models for non-english medical nlp. Journal of Biomedical Informatics, 145, 104478. 37625508 · DOI
  10. [14]He, J., Zhou, C, Ma, X, Berg-Kirkpatrick, T., & Neubig, G. (2021). Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366. · arXiv
  11. [17]Hoffmann, J, Borgeaud, S., Mensch, A, Buchatskaya, E, Cai, Trevor, R, Eliza, C, Diego de L., Hendricks, L.A., Welbl, J., Clark, A., et al.(2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. · arXiv
  12. [18]Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. · arXiv
  13. [20]Hu, Z., Lan, Y., Wang, L., Xu, W., Lim, E.P., Lee, R.K.W., Bing, L., & Poria, S. (2023). Llm-adapters: an adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933. · arXiv
  14. [21]Kojima, T., Gu, S.S,, Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. · arXiv
  15. [22]Köpf, Andreas, K., Yannic, von Rütte, Dimitri, A., Sotiris, T., Zhi-Rui, S., Keith, B., Abdullah, et al. (2023). Openassistant conversations - democratizing large language model alignment. arXiv: 2304.07327 [cs.CL]. · arXiv
  16. [26]Narayanan, Arvind, & Kapoor, Sayash. (2024). Ai snake oil: what artificial intelligence can do, what it can’t, and how to tell the difference. Princeton University Press.
  17. [27]Ollion, Étienne., Shen, Rubing, Macanovic, Ana, & Chatelain, Arnault. (2024). The dangers of using proprietary LLMs for research. Nature Machine Intelligence, 6 (1), 4–5. · DOI · Paper page
  18. [29]Pangakis, N., Samuel W., & Fasching, N. (2023). Automated annotation with generative ai requires validation. arXiv preprint arXiv:2306.00176. · arXiv
  19. [32]Rudin, Cynthia. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell, 1 (206), 215. PMC9122117 35603010
  20. [33]Sarti, G., Feldhus, N., Sickert, L., DerWal, Oskar Van, Nissim, M., & Bisazza, A. (2023). Inseq: an interpretability toolkit for sequence generation models. arXiv preprint arXiv:2302.13942. · arXiv
  21. [34]Schick, T., Dwivedi-Yu, J., Dessı, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2024). Toolformer: language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36.
  22. [35]Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2024). Hugginggpt: solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems 36.
  23. [38]Törnberg, P. (2023b). Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588. · arXiv
  24. [39]Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., & Bhosale, S., et al.(2023). Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  25. [41]Wang, Z., Wohlwend, J., & Lei, T. (2020). Structured pruning of large language models. In Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), 6151-6162. Online: Association for Computational Linguistics, November. 10.18653/v1/2020.emnlp-main.496. · DOI
  26. [42]Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, Adams W., Lester, B., Du, N., Dai, A.M., & Le, Q.V. (2022). Finetuned language models are zero-shot learners. arXiv: 2109.01652 [cs.CL]. · arXiv
  27. [43]Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., & Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv
  28. [45]Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023). Harnessing the power of llms in practice: a survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712. · arXiv
  29. [46]Yang, W., Li, C., Zhang, J., & Zong, C. (2023). Bigtrans: augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098. · arXiv
  30. [47]Zhang, B., Liu, Z., Cherry, C., & Firat, O. (2024). When scaling meets llm finetuning: the effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193. · arXiv
  31. [48]Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., & Smola, A. (2023). Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923. · arXiv

Cited by

Sorted by citations

Authors
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F.
Affiliations
  • University of Zurich (CH)
  • Allameh Tabataba'i University (IR)
  • Iran University of Science and Technology (IR)
Corresponding author
Alizadeh, Meysam
ORCID
Venue
Journal of Computational Social Science
Publisher
Springer Science and Business Media LLC
Published
2024-12-18
Volume
8
Issue
1
Pages
17-17
ISSN
  • 2432-2717 (ISSN-L)
  • 2432-2725
DOI
10.1007/s42001-024-00345-9
PMC
PMC11655591
OpenAlex
W4405552797
Cited by
80
References
13
Citations by year
2026: 46 · 2025: 32 · 2024: 2
FWCI
69.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Computational Social Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
Funders
  • H2020 European Research Council (883121)
  • University of Zurich
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
open-source large language models, text classification, textual annotation, fine-tuning, zero-shot learning, stance detection, topic classification, relevance classification, news articles, tweets, few-shot learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review