CoChat
Sign Up
PreprintCited by 114Computational and Text Analysis MethodsTopic ModelingSentiment Analysis and Opinion Mining

The DOI matches its DataCite record; 17 of the 17 references it cites were found in a registry or an index.

Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning

Open access
Alizadeh, Meysam, Kubli, Maël, Samei, Zeynab, Dehghani, Shirin, Zahedivafa, Mohammadmasiha, Bermeo, Juan Diego, Korobeynikova, Maria, Gilardi, Fabrizio·2023·arXiv
DOI 10.48550/arxiv.2307.02179arXiv 2307.02179
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT-3.5 and GPT-4, though still lagging behind fine-tuned GPT-3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.02179

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 23

  1. Binz, Marcel and Eric Schulz. 2023. 'Turning large language models into cognitive models.' arXiv:2306.03917 . · arXiv
  2. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al. 2020. 'Language models are few-shot learners.' Advances in neural information processing systems 33:1877-1901.
  3. Card, Dallas, Amber Boydstun, Justin H Gross, Philip Resnik and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) . pp. 438-444.
  4. Chung, Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma et al. 2022. 'Scaling instruction-finetuned language models.' arXiv:2210.11416 . · arXiv
  5. Dettmers, Tim, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer. 2024. 'Qlora: Efficient finetuning of quantized llms.' Advances in Neural Information Processing Systems 36.
  6. He, Junxian, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick and Graham Neubig. 2021. 'Towards a unified view of parameter-efficient transfer learning.' arXiv:2110.04366 . · arXiv
  7. Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark et al. 2022. 'Training compute-optimal large language models.' arXiv:2203.15556 . · arXiv
  8. Howard, Jeremy and Sebastian Ruder. 2018. 'Universal language model fine-tuning for text classification.' arXiv:1801.06146 . · arXiv
  9. Hu, Edward J, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2021. 'Lora: Low-rank adaptation of large language models.' arXiv:2106.09685 . · arXiv
  10. Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa. 2022. 'Large language models are zero-shot reasoners.' arXiv:2205.11916 . · arXiv
  11. Murdoch, Blake. 2021. 'Privacy and artificial intelligence: challenges for protecting health information in a new era.' BMC Medical Ethics 22(1):1-5.
  12. Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems , ed. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho and A. Oh. Vol. 35 Curran Associates, Inc. pp. 27730-27744.
  13. Pangakis, Nicholas, Samuel Wolken and Neil Fasching. 2023. 'Automated Annotation with Generative AI Requires Validation.' arXiv:2306.00176 . · arXiv
  14. Ray, Partha Pratim. 2023. 'ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.' Internet of Things and Cyber-Physical Systems .
  15. Schick, Timo, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda and Thomas Scialom. 2024. 'Toolformer: Language models can teach themselves to use tools.' Advances in Neural Information Processing Systems 36.
  16. Shen, Yongliang, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu and Yueting Zhuang. 2024. 'Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.' Advances in Neural Information Processing Systems 36.
  17. Törnberg, Petter. 2023 b . 'Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.' arXiv:2304.06588 . · arXiv
  18. Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale et al. 2023. 'Llama 2: Open foundation and fine-tuned chat models.' arXiv:2307.09288 . · arXiv
  19. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le and Denny Zhou. 2022. 'Chain of thought prompting elicits reasoning in large language models.' arXiv:2201.11903 . · arXiv
  20. Yang, Jingfeng, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin and Xia Hu. 2023. 'Harnessing the power of llms in practice: A survey on chatgpt and beyond.' arXiv:2304.13712 . · arXiv
  21. Yang, Wen, Chong Li, Jiajun Zhang and Chengqing Zong. 2023. 'BigTrans: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages.' arXiv:2305.18098 . · arXiv
  22. Zhang, Biao, Zhongtao Liu, Colin Cherry and Orhan Firat. 2024. 'When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method.' arXiv:2402.17193 . · arXiv
  23. Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis and Alex Smola. 2023. 'Multimodal chain-of-thought reasoning in language models.' arXiv:2302.00923 . · arXiv

Cited by

Sorted by citations

See all 114

Metadata

Authors
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.02179
arXiv
2307.02179
OpenAlex
W4383473003
Cited by
18
References
17
Citations by year
2025: 5 · 2024: 11 · 2023: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
open-source large language models, text classification, textual annotation, fine-tuning, zero-shot learning, few-shot learning, stance detection, topic classification, relevance classification, political science research, news articles, tweets, GPT-3.5, GPT-4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review