Prompt Stability Scoring for Text Annotation with Large Language Models
Open accessThe DOI matches its DataCite record; 54 of the 54 references it cites were found in a registry or an index; none of the 15 sources its claims cite is checked for support yet.
Abstract
Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package \texttt{promptstability} for its estimation. Using six different datasets and twelve outcomes, we classify $\sim$3.1m rows of data and $\sim$300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.
Overview
CoChat Summary
TL;DR
The paper proposes a Prompt Stability Score (PSS) that adapts intra- and inter-coder reliability to diagnose how sensitive LLM text annotations are to repeated runs and semantically similar prompt variants, and provides a Python package `promptstability`. Across six datasets and twelve outcomes, intra-prompt stability is often high (>0.8), but inter-prompt stability generally declines as prompts become more semantically distant, especially with ill-suited data or under-specified constructs.
Key findings
- The paper reports classifying ∼3.1m rows and ∼300m input tokens (abstract; Table 4 totals 3,144,022 rows and 306,390,955 input tokens). For most datasets, averaged intra-PSS is above 0.8 across runs; the shortened Synthetic data is among the least consistent.
- Inter-PSS generally decreases as paraphrase temperature increases. The least stable datasets across temperatures are those where data are ill-suited or the outcome construct is imprecisely defined; for some data-outcome pairings, PSS rapidly decreases even at low temperatures.
- Response formatting and filtering matter: MII and MII (Long) intra-PSS scores are lower on original annotations than on filtered data, and unique annotations increase as temperature increases.
- Cost is modest: redoing the analyses could cost 3 to 150 USD depending on sample size, and even a 2% subsample recovers basic trends for most datasets. gpt-4o-mini is more stable than gpt-3.5-turbo for the four worst outcomes, but Synthetic (Short) is about as stable.
Methods
- Six datasets (US Senator Tweets, NYT News Articles, UK Manifestos, US Politics Tweets, BES Survey Responses, ANES Voter Profiles), each with two outcome operationalizations = twelve outcomes; modifications target data complexity/suitability and construct validity/outcome complexity.
- Intra-PSS: the original prompt classifies the same maximum 500 rows 30 times, with cumulative Krippendorff’s Alpha and bootstrap confidence intervals (1,000 samples). Inter-PSS: PEGASUS generates semantically similar prompts across 25 temperatures from 0.1 to 5.0, with 10 variations per temperature, 3 iterations, and a maximum of 500 rows; scores are averaged within temperature.
- Main analyses use GPT-3.5 (`gpt-3.5-turbo`); appendix examples compare gpt-4o and DeepSeek-R1. The package is LM-agnostic.
Limitations
- Stated: only one paraphrasing engine is used; the approach does not adapt or iterate other model parameters such as the classification engine’s temperature or top p; the main analyses do not compare different LMs (though the software is LM-agnostic and appendix examples compare gpt-4o and DeepSeek-R1); and the tests do not use the exact original prompts, so they are not explicit replication tests of published findings.
- Stated: no specific cutoff is recommended; 0.8 is used as a heuristic, and parameter stability is left to future research.
- Evident: automated paraphrases at higher temperatures may be garbled or inaccurate, and format failures require filtering/balancing, which can reduce annotations and complicate interpretation; the main empirical results rely primarily on one LM (GPT-3.5), limiting cross-model generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
0 Valid · 8 Not valid · 9 Not checked · 56 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To take the Synthetic (short) data as an example, we see that for this challenging task where the data are not well suited to the annotation task, intra-PSS decreases quickly.”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 5 Results
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, when estimating the intra-PSS on the original annotations, both MII and MII (long) scores are relatively a lot lower than their scores when estimating on the filtered data.”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 5 Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that even for the sizeable number of iterations, temperatures, and variations that we conducted, the cost (here for using gpt-3.5-turbo is minimal.”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 5.1 How much does this all cost?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see from Figure 3 that even for 2% of the sample, we are able to recover the basic trends for most datasets.”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 5.1 How much does this all cost?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, it is clear that when the annotation task is particularly challenging-as is the case with the shortened Synthetic data-the intra-PSS scores are among the least consistent.”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 5 Results
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
28 more claims
Create a free account to see all 73 claims and to verify the 11 references still open.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 33
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The potential lack of replicability is another hurdle that lowers social scientists' enthusiasm to adopt LLMs in their research (Barrie, Palaiologou, & Törnberg, 2024).”
From Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research (2026) · cites this paper as Barrie, Palaiologou, & Törnberg, 2024
In the citing paper
“A recent work conducted by Barrie et al. (2024) shows that the prompt stability of outputs generally decreases with increasing temperature due to the degradation in the quality of prompts.”
From Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research (2026) · cites this paper as Barrie et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Prompt Stability Scoring for Text Annotation with Large Language Models > A10
“Your task is to read each article and label its overall sentiment as positive or negative .”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > A10
“[ Respond 0 for negative , 1 for positive , and 2 for neutral .”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 4.2.2 Inter-prompt stability
“We also provide results for poor performing tests using an updated OpenAI model (gpt-4o).”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 5.3 Are other LMs more stable?
“and Synthetic (Short) and re-estimate the inter-PSS using gpt-40-min.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Prompt Stability Scoring for Text Annotation with Large Language Models > A10
“s t a n c e _ l o n g _ w i t h i n .csv The text provided come from some tweets about Donald Trump .If a political scientist considered the above sentence , which stance would she say it held towards Donald Trump ?[ Respond 0 for negative , and 1 for positive .Respond nothing else .]”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > A10
“s t a n c e _ l o n g _ w i t h i n .csv The text provided come from some tweets about Donald Trump .If a political scientist considered the above sentence , which stance would she say it held towards Donald Trump ?[ Respond 0 for negative , 1 for positive , and 2 for none .Respond nothing else .]”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 4.1 Data
“We modify the data complexity or suitability in the US Senator Tweets, NYT News Articles, US Politics Tweets, BES Survey Responses, and ANES Voter Profiles;”
§ Prompt Stability Scoring for Text Annotation with Large Language Models > 4.1 Data
“US Senator Tweets van Vliet et al. (2020) • Binary: Republican/Democrat • Binary: Populist/Not Populist • Tweets (Rep.Dem.) • Tweets (Populism) • Construct validity/Outcome complexity”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2001.07676 · Cited by 1,944
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
Research works printed but not linked: 4
- Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34: 11054-11070, 2021.
- Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, Marta Miori, and Phillip Schmedeman. Do ais know what the most important issue is? using language models to code open-text social survey responses at scale. Research & Politics, 11(1):20531680241231468, 2024.
- Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems, 22, 2009.
- DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501. 12948.
Cited by
Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text AnnotationPreprint
Baumann, J., Röttger, P., Urman, A., Wendsjö, A., Plaza-del-Arco, F. M., Gruber, J. B., & Hovy, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.08825 · Cited by 107
Machine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 88
Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Lin, H., & Zhang, Y. (2026) · Social Science Computer Review · doi:10.1177/08944393251366243 · Cited by 33
Carammia, M., Iacus, S. M., & Porro, G. (2026) · Social Science Computer Review · doi:10.1177/08944393261485320 · Cited by 19
Interactive Data Harmonization with LLM Agents: Opportunities and ChallengesPreprint
Santos, A. S. R., Pena, E. H. M., López, R., & Freire, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.07132 · Cited by 19
Metadata
- Authors
- Barrie, C., Palaiologou, E., & Törnberg, P.
- Affiliations
- New York University (US)
- University of Amsterdam (NL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2407.02039
- OpenAlex
- W4400377925
- Semantic Scholar
- a523a7b890e720634bf63aea6df63d7e47c731ef
- Cited by
- 44
- References
- 42
- Citations by year
- 2026: 25 · 2025: 17 · 2024: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- prompt engineering, large language models, textual annotation, intercoder reliability, reproducibility, text classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Prompt Stability Scoring for Text Annotation with Large Language Models. https://app.cochat.ai/papers/10.48550/arxiv.2407.02039
- The paper
- Barrie, C., Palaiologou, E., & Törnberg, P. (2024). Prompt Stability Scoring for Text Annotation with Large Language Models. arXiv. https://doi.org/10.48550/arxiv.2407.02039
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.