CoChat

Prompt Stability Scoring for Text Annotation with Large Language Models

Open access

The DOI matches its DataCite record; 54 of the 54 references it cites were found in a registry or an index; none of the 15 sources its claims cite is checked for support yet.

Abstract

Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package \texttt{promptstability} for its estimation. Using six different datasets and twelve outcomes, we classify $\sim$3.1m rows of data and $\sim$300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

Overview

CoChat Summary

TL;DR

The paper proposes a Prompt Stability Score (PSS) that adapts intra- and inter-coder reliability to diagnose how sensitive LLM text annotations are to repeated runs and semantically similar prompt variants, and provides a Python package `promptstability`. Across six datasets and twelve outcomes, intra-prompt stability is often high (>0.8), but inter-prompt stability generally declines as prompts become more semantically distant, especially with ill-suited data or under-specified constructs.

Key findings

  • The paper reports classifying ∼3.1m rows and ∼300m input tokens (abstract; Table 4 totals 3,144,022 rows and 306,390,955 input tokens). For most datasets, averaged intra-PSS is above 0.8 across runs; the shortened Synthetic data is among the least consistent.
  • Inter-PSS generally decreases as paraphrase temperature increases. The least stable datasets across temperatures are those where data are ill-suited or the outcome construct is imprecisely defined; for some data-outcome pairings, PSS rapidly decreases even at low temperatures.
  • Response formatting and filtering matter: MII and MII (Long) intra-PSS scores are lower on original annotations than on filtered data, and unique annotations increase as temperature increases.
  • Cost is modest: redoing the analyses could cost 3 to 150 USD depending on sample size, and even a 2% subsample recovers basic trends for most datasets. gpt-4o-mini is more stable than gpt-3.5-turbo for the four worst outcomes, but Synthetic (Short) is about as stable.

Methods

  • Six datasets (US Senator Tweets, NYT News Articles, UK Manifestos, US Politics Tweets, BES Survey Responses, ANES Voter Profiles), each with two outcome operationalizations = twelve outcomes; modifications target data complexity/suitability and construct validity/outcome complexity.
  • Intra-PSS: the original prompt classifies the same maximum 500 rows 30 times, with cumulative Krippendorff’s Alpha and bootstrap confidence intervals (1,000 samples). Inter-PSS: PEGASUS generates semantically similar prompts across 25 temperatures from 0.1 to 5.0, with 10 variations per temperature, 3 iterations, and a maximum of 500 rows; scores are averaged within temperature.
  • Main analyses use GPT-3.5 (`gpt-3.5-turbo`); appendix examples compare gpt-4o and DeepSeek-R1. The package is LM-agnostic.

Limitations

  • Stated: only one paraphrasing engine is used; the approach does not adapt or iterate other model parameters such as the classification engine’s temperature or top p; the main analyses do not compare different LMs (though the software is LM-agnostic and appendix examples compare gpt-4o and DeepSeek-R1); and the tests do not use the exact original prompts, so they are not explicit replication tests of published findings.
  • Stated: no specific cutoff is recommended; 0.8 is used as a heuristic, and parameter stability is left to future research.
  • Evident: automated paraphrases at higher temperatures may be garbled or inaccurate, and format failures require filtering/balancing, which can reduce annotations and complicate interpretation; the main empirical results rely primarily on one LM (GPT-3.5), limiting cross-model generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.02039

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Combined intra-prompt stability scores ordered by average score.
Figure 1 Combined intra-prompt stability scores ordered by average score.CC-BY · From the PDF
Figure 2 Combined inter-prompt stability scores, ordered by average score.
Figure 2 Combined inter-prompt stability scores, ordered by average score.CC-BY · From the PDF
Figure 3 Combined inter-prompt stability scores for random subsamples.
Figure 3 Combined inter-prompt stability scores for random subsamples.CC-BY · From the PDF
Figure 4 Combined inter-prompt stability scores for lowest inter-PSS outcomes in main analyses.
Figure 4 Combined inter-prompt stability scores for lowest inter-PSS outcomes in main analyses.CC-BY · From the PDF
Figure 5 Combined intra- and inter-prompt stability scores for DeepSeek-R1 and GPT-4o.
Figure 5 Combined intra- and inter-prompt stability scores for DeepSeek-R1 and GPT-4o.CC-BY · From the PDF
Figure B.1: Combined original intra-prompt stability scores.
Figure B.1: Combined original intra-prompt stability scores.CC-BY · From the PDF
Figure B.2: Unique annotations by dataset and iteration.
Figure B.2: Unique annotations by dataset and iteration.CC-BY · From the PDF
Figure B.3: Combined original inter-prompt stability scores.
Figure B.3: Combined original inter-prompt stability scores.CC-BY · From the PDF
Figure B.4: Unique annotations by dataset and temperature.
Figure B.4: Unique annotations by dataset and temperature.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 8 Not valid · 9 Not checked · 56 Own results and interpretations

28 more claims

Create a free account to see all 73 claims and to verify the 11 references still open.

See all claims — free account
  • It may state two separate claims as one. 12
  • Probably not a substantial claim. 33
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 54 references

Research works printed but not linked: 4

  1. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34: 11054-11070, 2021.
  2. Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, Marta Miori, and Phillip Schmedeman. Do ais know what the most important issue is? using language models to code open-text social survey responses at scale. Research & Politics, 11(1):20531680241231468, 2024.
  3. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems, 22, 2009.
  4. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501. 12948.

Cited by

See all 44

Metadata

Authors
Barrie, C., Palaiologou, E., & Törnberg, P.
Affiliations
  • New York University (US)
  • University of Amsterdam (NL)
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4400377925
Cited by
44
References
42
Citations by year
2026: 25 · 2025: 17 · 2024: 2
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
prompt engineering, large language models, textual annotation, intercoder reliability, reproducibility, text classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Prompt Stability Scoring for Text Annotation with Large Language Models. https://app.cochat.ai/papers/10.48550/arxiv.2407.02039
The paper
Barrie, C., Palaiologou, E., & Törnberg, P. (2024). Prompt Stability Scoring for Text Annotation with Large Language Models. arXiv. https://doi.org/10.48550/arxiv.2407.02039

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.