Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Recent studies have demonstrated promising potential of ChatGPT for various text annotation and classification tasks. However, ChatGPT is non-deterministic which means that, as with human coders, identical input can lead to different outputs. Given this, it seems appropriate to test the reliability of ChatGPT. Therefore, this study investigates the consistency of ChatGPT’s zero-shot capabilities for text annotation and classification, focusing on different model parameters, prompt variations, and repetitions of identical inputs. Based on the real-world classification task of differentiating website texts into news and not news, results show that consistency in ChatGPT’s classification output can fall short of scientific thresholds for reliability. For example, even minor wording alterations in prompts or repeating the identical input can lead to varying outputs. Although pooling outputs from multiple repetitions can improve reliability, this study advises caution when using ChatGPT for zero-shot text annotation and underscores the need for thorough validation, such as comparison against human-annotated data. The unsupervised application of ChatGPT for text annotation and classification is not recommended.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Absent from recent contributions, however, is any equivalent concern with reliability scores when using LMs [Reiss, 2023, Ollion et al., 2023, Sclar et al., 2023].”
From Prompt Stability Scoring for Text Annotation with Large Language Models (2024) · cites this paper as Reiss, 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2302.06476 · Cited by 869
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT
Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023) · arXiv · doi:10.48550/arxiv.2302.10198 · Cited by 146
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2023) · arXiv · doi:10.48550/arxiv.2304.00228 · Cited by 115
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81
Cited by
Ali, F., Yasar, B., Ali, L., & Dogan, S. (2023) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2023.103588 · Cited by 185
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76
Can ChatGPT Decipher Fedspeak?
Hansen, A. L., & Kazinnik, S. (2023) · doi:10.2139/ssrn.4399406 · Cited by 76
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63
Metadata
- Authors
- Reiss, M.
- Publisher
- Center for Open Science
- Published
- 2023-04-06
- OpenAlex
- W4362700315
- Cited by
- 94
- References
- 7
- Type
- posted content
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.