ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

Zhuo, Jingming; Zhang, Songyang; Fang, Xinyu; Duan, Haodong; Lin, Dahua; Chen, Kai

doi:10.48550/arxiv.2410.12405

PreprintCited by 78Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

Open access

Zhuo, Jingming, Zhang, Songyang, Fang, Xinyu, Duan, Haodong, Lin, Dahua, Chen, Kai·2024·arXiv

DOI 10.48550/arxiv.2410.12405arXiv 2410.12405

Open this paper in CoChat — free account Open PDF

Abstract

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: https://github.com/open-compass/ProSA .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.12405

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Meaning-preserving changes (spacing, punctuation, example ordering, case) cause substantial performance shifts.
“Zhuo et al. (2024); Sclar et al. (2024); Andersson et al. (2025) show that meaning-preserving changes (spacing, punctuation, example ordering, case) cause substantial performance shifts.”
From On Randomness in Agentic Evals (2026) · cites this paper as Zhuo et al. (2024)
Section 4.1.3 says the rewritten prompts were checked for semantic similarity, and Section 4.2 reports sensitivity to those variants on challenging questions, while finding relative robustness on simpler questions. The paper does not establish that the specific changes listed—spacing, punctuation, example ordering, and case—caused substantial performance shifts. Page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measured
A key challenge in prompt-based website generation is that user-provided prompts often lack essential functional or structural requirements of a complete website.
“A key challenge in prompt-based website generation is that user-provided prompts often lack essential functional or structural requirements of a complete website [75, 78].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [78]
The Introduction (page unavailable) discusses how variations in prompt wording affect LLM outputs, and §2.1 defines task requirements and different expressions of them. Neither addresses website generation or says that user prompts often omit essential functional or structural requirements; searches for website generation and incomplete requirements found no such passage. (LLM, conf=0.76)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“In the era of LLMs, few-shot plays a critical role in enabling LLMs to follow specific formats and improve their performance (Brown et al., 2020).”
Background
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1× · see citing works
“First, we used the "all-MiniLM-L6-v2" model from Sentence Transform-ers (Reimers and Gurevych, 2019) to calculate the BERTScore (Zhang et al., 2020) (F1 version) be-tween the original and rewritten prompts.”
Method
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 1× · see citing works
“First, we used the "all-MiniLM-L6-v2" model from Sentence Transform-ers (Reimers and Gurevych, 2019) to calculate the BERTScore (Zhang et al., 2020) (F1 version) be-tween the original and rewritten prompts.”
Method
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,805 · see citing works
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · cited 1× · see citing works
“By training on large-scale corpora, LLMs have shown impressive capabilities in multiple tasks (Zhao et al., 2023; Min et al., 2023).”
Background
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 1× · see citing works
“These specific tasks include, but are not limited to, reading comprehension (Sakaguchi et al., 2021; Mostafazadeh et al., 2017), mathematical problem solving (Cobbe et al., 2021; Hendrycks et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021).”
Background
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 1× · see citing works
“These specific tasks include, but are not limited to, reading comprehension (Sakaguchi et al., 2021; Mostafazadeh et al., 2017), mathematical problem solving (Cobbe et al., 2021; Hendrycks et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021).”
Background
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 1× · see citing works
“These specific tasks include, but are not limited to, reading comprehension (Sakaguchi et al., 2021; Mostafazadeh et al., 2017), mathematical problem solving (Cobbe et al., 2021; Hendrycks et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021).”
Background
LSDSem 2017 Shared Task: The Story Cloze Test
Mostafazadeh, N., Roth, M., Louis, A. P., Chambers, N., & Allen, J. F. (2017) · LSDSem@EACL · doi:10.18653/v1/w17-0906 · Cited by 129 · cited 1× · see citing works
“These specific tasks include, but are not limited to, reading comprehension (Sakaguchi et al., 2021; Mostafazadeh et al., 2017), mathematical problem solving (Cobbe et al., 2021; Hendrycks et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021).”
Background
State of What Art? A Call for Multi-Prompt LLM Evaluation
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00681 · Cited by 113 · cited 1× · see citing works
“(Mizrahi et al., 2024) demonstrated that model robustness leads to cherry-picking of model performance.”
Background
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023) · arXiv · doi:10.48550/arxiv.2310.11324 · Cited by 52 · cited 2×
“Previous study (Zhu et al., 2023; Pezeshkpour and Hruschka, 2024; Sclar et al., 2024) showed that LLMs are sensitive to prompts, and that perturbing the prompt can cause a significant variation in the performance of models.”
Background
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · cited 3× · see citing works
“Previous research (Zhu et al., 2023; Pezeshkpour and Hruschka, 2024) compares different prompt templates within datasets (composed of instances) to analyze the prompt sensitivity of LLMs, which is done by examining the shifts in the LLMs’ scores across different prompt templates within the dataset.”
BackgroundMethod
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024) · arXiv · doi:10.48550/arxiv.2404.04475 · Cited by 17 · cited 1× · see citing works
“Recently, with the improvement in the objective capabilities of LLMs, subjective evaluation, which aims to assess the alignment of generated responses with human needs, such as LC AlpacaEval 2.0 (Dubois et al., 2024), has received increasing attention.”
BackgroundInfluential
A Survey of Confidence Estimation and Calibration in Large Language ModelsPreprint
Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., & Gurevych, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.08298 · Cited by 4 · cited 1× · see citing works
“We use Token Probabilities, a widely employed method for measuring a model’s confidence in its generated content (Schuster et al., 2022; Geng et al., 2023), to calculate the model’s decoding confidence.”

Flagged first, then by citations

9 of the 24 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of op- tions in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006-2017, Mexico City, Mexico. Asso- ciation for Computational Linguistics. · DOI · Paper page
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM, 64(9):99-106.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Ad- vances in Neural Information Processing Systems, 35:17456-17472.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. · arXiv
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023. Lmsys- chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998. · arXiv
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528. · arXiv

Sorted by citations

Benchmarking Prompt Sensitivity in Large Language Models
Razavi, A., Soltangheis, M., Arabzadeh, N., Salamat, S., Zihayat, M., & Ebrahim Bagheri, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_29 · Cited by 39 · see citing works
Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
Khan, A., Casper, S., & Hadfield-Menell, D. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732147 · Cited by 16 · see citing works
A study of calibration as a measurement of trustworthiness of large language models in biomedical natural language processing
de Oliveira, R., Garber, M., Gwinnutt, J. M., Rashidi, E., Hwang, J.-H. (., Gilmour, W., Nanavati, J., Abidine, K. Z. E., & Mack, C. D. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf058 · Cited by 17 · see citing works
Design coco-pilot: QFD as decision logic for conflict-aware human–LLM collaborative product conceptual design
Huang, R., Yang, T., Feng, S., & Li, F. (2025) · Journal of Engineering Design · doi:10.1080/09544828.2025.2578826 · Cited by 13 · see citing works
Large language models show both individual and collective creativity comparable to humans
Sun, L., Yuan, Y., Yao, Y., Li, Y., Zhang, H., Xie, X., Wang, X., Luo, F., & Stillwell, D. J. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101870 · Cited by 18 · see citing works
Swedish Medical LLM Benchmark: development and evaluation of a framework for assessing large language models in the Swedish medical domain
Moëll, B., Farestam, F., & Beskow, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1557920 · Cited by 12 · see citing works
Language models for drug–drug interactions: current applications, pitfalls, and future directions
Meslamani, A. Z. A., & Hajal, A. A. (2025) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1080/17425255.2025.2551724 · Cited by 12 · see citing works
Knowledge integration for physics-informed symbolic regression using pre-trained large language models
Taskin, B., Xie, W., & Lazebnik, T. (2026) · Scientific Reports · doi:10.1038/s41598-026-35327-6 · Cited by 10 · see citing works
“Table 3 summarizes the results of the noise experiment, dividing between noise addition to the source features, the target feature, for the cases of 1% to 5% standard deviation noise addition.”
RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
Cheng, S., Li, J., Wang, H., & Ma, Y. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747741 · Cited by 7 · see citing works
Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences
Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00843-4 · Cited by 13 · see citing works
“Kim and Lee 77 focused on providing accurate estimates for narrow subpopulations or single individuals, making a case that fine-tuning can help LLMs provide more accurate estimates.”
Looking Under the Hood: How LLMs Attempt Political Persuasion and Microtargeting
Lyman, A., Busby, E. C., Argyle, L. P., Gubler, J. R., Hepner, B., & Wingate, D. (2026) · Chinese Political Science Review · doi:10.1007/s41111-026-00336-3 · Cited by 5 · see citing works
Vision language models versus machine learning models performance on polyp detection and classification in colonoscopy images
Khalafi, M. A., Safavi‐Naini, S. A. A., Salehi, A., Naderi, N., Alijanzadeh, D., Moghadam, P. K., Kavousi, K., Golestani, N., Shahrokh, S., Fallah, S., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-29566-2 · Cited by 5 · see citing works
Retrospective modeling study of ChatGPT (GPT 4) warfarin dose adjustment in patients with INRs outside the therapeutic range
Tezcan, H., Tunçez, A., Özen, Y., Yalçın, M. U., & Gürses, K. M. (2026) · Digital Health · doi:10.1177/20552076251412985 · Cited by 4 · see citing works
“20 – 22 Accordingly, our concordance estimates should be interpreted as re ﬂ ecting performance under a pre-speci ﬁ ed, directive prompt intended to approximate standard maintenance dosing logic.”
VAP3: Variation-Aware Prompt Performance Prediction
Arabzadeh, N., & Ebrahim Bagheri, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730264 · Cited by 3 · see citing works
Explainable pedagogical alignment: auditing LLMs with learning theories
Figaredo, D. D., Vindel, J. L. F., & Puerta, L. G. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13857-2 · Cited by 3 · see citing works
LLM-Based Automated Hallucination Detection in Multilingual Customer Service RAG Applications
Patel, N. R., Mouratidis, H., & Zhi, K. N. K. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96235-6_26 · Cited by 3 · see citing works
Dropping Anchor or Chasing the Horizon? Theoretical and Practical Challenges for Personalized AI Advisors
Lang, B. H. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00990-6 · Cited by 3 · see citing works
Multi-level prompting: Enhancing model performance through hierarchical instruction integration
Son, G., & Kim, M. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113545 · Cited by 2 · see citing works
SMALLM: a local small model augmented a cloud-based large language model for chinese named entity recognition in low-resource industries
Yang, J., Yang, Z., Chengke, W., Guo, Y., Li, X., & Lin, J. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02074-6 · Cited by 2 · see citing works
An empirical study of testing practices in open source AI agent frameworks and agentic applications
Hasan, M. M., Li, H., Fallahzadeh, E., Rajbahadur, G. K., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10857-9 · Cited by 11 · see citing works
Enhancing Clinical Trial Patient Matching through Knowledge Augmentation and Reasoning with Multi-Agent
Shi, H., Zhang, J., & Zhang, K. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00111 · Cited by 8 · see citing works
“First, we remove the knowledge augmentation module to assess the reasoning module alone (MAKAR-Reason, equivalent to a CoT) using the original criteria described in Appendix A.”
Influential
On Randomness in Agentic Evals
Bjarnason, B. H., Silva, A., & Monperrus, M. (2026) · arXiv · doi:10.48550/arxiv.2602.07150 · Cited by 1 · see citing works
Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs
Freedman, G., & Toni, F. (2025) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.38.1.138892 · Cited by 1 · see citing works
“Due to the high variation in performance depending on the formulation of the prompt (Zhuo et al. 2024), we form a validation set from 468 samples (117 of each variety of claim) in our dataset.”
Using Semantically Unrelated and Opposite Terms for In-Context Learning: A Case Study in Identifying Political Aversion in Tweets
Wu, P. Y. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717913 · Cited by 1 · see citing works
“Most of the works in this area have focused on how results from generative LLMs can vary when using different but semantically equivalent prompts [see, e.g., 1, 9, 13, 15, 22–24].”
Passenger Immersive Experiences in Autonomous Vehicles: A Comprehensive Review and Proposed Framework
Zhang, Y., Liu, B., Pike, M., Wang, C., & Sun, X. S. (2025) · Journal of Advanced Transportation · doi:10.1155/atr/4874071 · Cited by 2 · see citing works
“Additional work is needed to document how specifc prompts were crafted and examine multiple crafted prompts on the same input for prompt sensitivity testing [106, 107].”
Exploring Personality Structure Through LLM Agent
Hashimoto, Y., & Oshio, A. (2025) · Psychological Test Adaptation and Development · doi:10.1027/2698-1866/a000114 · Cited by 1 · see citing works
Evaluating Large Language Models as Judicial Decision-Makers
Gazal‐Ayal, O., Elyoseph, Z., & Solomon, A. (2026) · Justice Quarterly · doi:10.1080/07418825.2026.2618254 · Cited by 3 · see citing works
“Prior work shows that LLMs can be sensitive to such variations, occasionally yielding divergent outputs even when the underlying facts remain constant (Zhuo et al., 2024).”
A framework for evaluation of large language models in essay assessment: Reliability, alignment, and causal reasoning
Liu, T., Ye, L., & Yan, W. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100565 · Cited by 3 · see citing works
FairVLM: Enhancing Fairness and Prompt Sensitivity in Vision Language Models for Medical Image Segmentation
Rahman, M. M., Rahman, S., Bhatt, S., & Faezipour, M. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00719 · Cited by 1 · see citing works
“Additionally, prompt sensitivity, such as minor wording changes in radiology reports, can lead to inconsistent segmentation results [19, 46].”
Prompt anxiety and the algorithmic politics of uncertainty
Berry, D. M. (2026) · AI & Society · doi:10.1007/s00146-026-03093-8 · Cited by 2 · see citing works
“I argue that the technical architecture of these systems actively encourages a paranoid style through what developers call “prompt sensitivity” (Zhuo et al. 2024).”
The shift towards auto-labeling in natural language processing: a literature review on large language model-based methods
Nabil, H. A. M., Lutfi, S. L., & Mulki, H. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3878 · Cited by 1 · see citing works
The Fragility of Moral Judgment in Large Language Models
van Nuenen, T., & Sachdeva, P. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806447 · Cited by 1 · see citing works
How Does Prompt Anchoring Affect Large Language Model Outputs?
Kim, E. (2026) · Publications · doi:10.3390/publications14030043 · Cited by 1 · see citing works
“Research on prompt design shows that prompt engineering plays a central role in shaping LLM output quality, and that model behavior is highly sensitive to prompt variation (Chatterjee et al., 2024; Lu et al., 2024; Zhuo et al., 2024).”
Large language model accuracy in inhaler technique counselling for asthma and COPD: A comparison of free and paid models across ten devices
KOÇ, A., Sağun, F., Öğe, N., Avcil, S., Çelik, A. T., Takes, M. T.-L., & Sunay, B. (2026) · Digital Health · doi:10.1177/20552076261478506 · Cited by 1 · see citing works
“…of up to 45% between semantically equivalent, 29 and a more comprehensive prompt-sensitivity framework has shown that larger models exhibit greater robustness than smaller ones and that sensitivity is most pronounced for complex reasoning tasks 30 Seventh, the evaluation used single-turn prompts.”
Large language models unlock the ecology of species interactions
Zou, H., Yang, X., Hajamaideen, T. H., Stein, O., Beltran, R. S., Freeman, B. G., Lindquist, M., Miller, E. T., Mengarelli, S., Probst, C. M., et al. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2602244123 · Cited by 2 · see citing works
Is artificial intelligence revolutionising agricultural sciences yet? An AI ‐based bibliometric analysis
Kozak, M. (2025) · Annals of Applied Biology · doi:10.1111/aab.70010 · Cited by 1 · see citing works
Prompt Fairness: Sub-group Disparities in LLMs
Zhong, M., Teku, N., & Tandon, R. (2025) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf67917.2025.11443738 · Cited by 1 · see citing works
“However, recent studies have shown that semantically equivalent prompts can elicit divergent answers from state-of-the-art LLMs [1] [2], raising concerns about robustness and fairness in real-world applications.”
Personas Shift Clinical Action Thresholds in Large Language ModelsPreprint
Klang, E., Gorenshtein, A., Omar, M., & Nadkarni, G. N. (2026) · medRxiv · doi:10.64898/2026.01.01.26343302 · Cited by 1 · see citing works
Integrating Semantic Retrieval, LLM-based Refinement, and Structured Expert Curation for Scalable AOP Gene MappingPreprint
Schaffert, A., Fratello, M., Kangas, K., Maia, M. T., del Giudice, G., Möbus, L., Accardi, C., Al-Abdulraheem, Z., Campini, L., Galardo, F., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.25.734475 · Cited by 1 · see citing works
“…frameworks for analyzing prompt sensi?vity demonstrate that LLM robustness varies across tasks and datasets, and that instance-level prompt varia?on can materially aﬀect both objec?ve performance and subjec?ve evalua?ons, underscoring the need for systema?c prompt assessment (Zhuo et al., 2024).”
A Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1 · see citing works
Generalist large language models in a specialized world: Evidence from the Italian national medical education pathway
Buonocore, T. M., Russo, A., Mingarelli, D., & Parimbelli, E. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001363 · Cited by 3 · see citing works
“Previous studies show that the performance of LLMs might be affected by the format used to build the input prompt, especially for relatively small LLMs [25].”
Development of a Virtual Assistant Based on LLMs for the Knowledge Domain in Biomedical Metrology
Carreon, Y., & Chicchón, M. (2025) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v21i09.55653
A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System ChallengesPreprint
Raja, R., Vats, A., Vats, A., & Majumder, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5359269 · Cited by 5 · see citing works
Rethink Your Mental Model in the Age of Generative AI: A Triadic Framework for Human-AI CollaborationPreprint
Saßmannshausen, T. M., & Wagener, S. (2026) · Qeios · doi:10.32388/gag6kd
Rethink Your Mental Model in the Age of Generative AI: A Triadic Framework for Human-AI CollaborationPreprint
Saßmannshausen, T. M., & Wagener, S. (2026) · Qeios · doi:10.32388/gag6kd.2
Making Large Language Models More Flexible to Changing Contingencies through Tracking Induction
Ruiz, F. J., & Martínez-Carrillo, E. (2026) · The Psychological Record · doi:10.1007/s40732-026-00698-9
Corporate Communication, Agentic AI, and Prompt Injections
Langenbucher, K. C., & Bauer, K. (2026) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0141
ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
Li, B. R., Huang, W., Qiu, Y., Guo, Y., Wang, H., Li, Z., Peng, J., Ma, Z., Chen, X., & Yu, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462799 · Cited by 1 · see citing works
Human-AI collaboration in classroom video analysis: evaluating the feasibility and usability of multimodal large language models for nonverbal behavior recognition
Zhou, Y., Kang, J., & Nguyen, H. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2662406
Exploratory Study on LLMs for Test Amplification in Solidity
Handenhoven, J. V., Vanhove, B., Beyazıt, M., Kılınççeker, O., & Demeyer, S. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00010
Decomposing Complexity: A Difficulty-Aware Multi-agent Framework for Open-Domain Knowledge Graph Construction
Chen, H., Liu, Y., Li, C., & Hu, W. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0378-9_20
Fostering Organizational Collective Intelligence From Human-Generative AI Interaction
Hino, S., Nakata, T., Chen, S., Saiki, S., & Nakamura, M. (2026) · International Journal of Software Innovation · doi:10.4018/ijsi.411903
The communication of risks associated with GLP-1 receptor agonists: An AI-powered content analysis of TikTok videos (Preprint)Preprint
Almenara, C. A., Anastasiadou, D. T., & Flint, S. W. (2026) · doi:10.2196/preprints.102913
Advancing Grounded Multimodal NER via Self-reflective Prompt Refinement and Visual Noise Mitigation
Liang, Q., Zhou, W., Cao, S., Zhang, W., & Zheng, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8525-0_21
Interpretable Legal Similarity: From Embeddings to Obligations
Roegiest, A., & Trippas, J. R. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820406
Hallucinations in LLMs: a lifecycle-based survey of causes, detection, mitigation, and prevention
Lamba, N., Tiwari, S., & Gaur, M. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01214-6 · Cited by 1 · see citing works
“• Perplexity and Entropy-Based Detection: Here, n is the total number of neurons, σ 2 l is the variance of activations in layer l , µ l is the mean activation across neurons in layer l , and a ( l ) i is the activation of the i th neuron in that layer.”
Do Prompt-Level Empathy Instructions Influence User Experience? Evidence From A Controlled Chatbot Study
Bieberstein, A., Schnitzer, B. L., Gampe, S., & Korn, O. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816221
Do influence tactics matter? investigating prompt framing effects in LLM code generation
Deaconu, A., Gupta, A., Basha, M., Haydu, N., & Rodríguez-Pérez, G. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10934-z
“Prior research on LLM prompt sensitivity suggests that models are generally more susceptible to pragmatic prompt variations in tasks involving complex reasoning or open-ended generation, while they tend to be more stable on tasks with constrained and unambiguous solutions [82].”
Semantically Enriching Investor Micro-blogs for Opinion-Aware Emotion Analysis: A Practical Approach
Negi, G., Zayed, O., Serrano, M., & Buitelaar, P. (2026) · IEEE International Conference on Smart Computing Workshops and Other Affiliated Events (SMARTCOMP Companion) · doi:10.1109/smartcomp-companion70724.2026.00045
“The opinions to augment our dataset are generated following the pipeline introduced by [28] that follows the paradigm of “program-ming LLM” [29] rather than using naive prompts, which are known to be sensitive [30].”
Style and Stance Alignment in Reddit Discourse
Jordão, S. A. A., Santos, C. S., dos Santos, F. C. V., & Dantas, M. J. P. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00321
“Because the stance units and labels are produced by an LLM, the empirical object of the study is model annotated stance alignment rather than manually verified human ground truth; this distinction follows recent methodological cautions about using LLMs as annotators [26]–[30].”
Global Trends in Research Ethics in the Context of Artificial Intelligence
Resulbegoviq, H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1232-1.ch012
Proof Schemes Used and Owned by Artificial Intelligence Chatbots
Cihan, F. (2026) · Disiplinlerarası Eğitim Araştırmaları Dergisi · doi:10.57135/jier.1894348
A progressive verification method for false information detection using LLMs in human-AI hybrid information environments
Tao, T., & Li, Y. (2026) · Aslib Journal of Information Management · doi:10.1108/ajim-11-2025-0940
Large language model-based evaluation of the impact of gender in medical research
Yao, M. S. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag173
Generative AI Technologies, Techniques and Tensions: A Primer
Behrens, J. T. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_2 · Cited by 1 · see citing works
Координирано използване на големи езикови модели в академичната работа
Marinov, V. (2026) · Media and Language Journal · doi:10.58894/dvzte852
Prompt Sensitivity or Evaluation Artifact? A Task-Aware Analysis for Large Language Models
Muthukumarana, S., Wijenayake, B., Godaliyadda, R., & Ekanayake, P. (2026) · Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon71835.2026.11691277
“At the instance level, Zhuo et al. [7] propose ProSA, measuring within-instance output variance via semantic similarity, but do not decompose the disagreement between heuristic and judge evaluation methods.”
AI information literacy in knowledge work: a pathway from tool awareness to responsible information practice
Levkovich, I., & Alon, L. (2026) · Journal of Documentation · doi:10.1108/jd-05-2026-0295
Beyond accuracy: an educational benchmarking study of task fragility and reasoning stability of large language models on dermatology board-style questions
Atılan, A. U., & Çetin, N. (2026) · Cutaneous and Ocular Toxicology · doi:10.1080/15569527.2026.2733364 · Cited by 1 · see citing works
Dynamic Capabilities for Brand Equity in the Agentic Age
Buttenberg, K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9589-0.ch001
Developing a Temperament- and Character-Based System for Virtual Patient Generation and Response Comparison
Choi, Y.-M., Hur, S., & Kim, S.-H. (2026) · Journal of Digital Contents Society · doi:10.9728/dcs.2026.27.9.2573
Capability Thresholds and Sanitisation Blind Spots in LLM-Assisted Vulnerability Detection: A Factorial Prompt Study on a Reconstructed PHP Benchmark
Lin, H.-L., Kao, H.-E., Pi, S., & Li, K.-C. (2026) · IEEE Access · doi:10.1109/access.2026.3741095
“A systematic evaluation by Zhuo et al. [22] of prompt sensitivity across models and tasks further reports that small variations in prompt phrasing produce measurable accuracy differences and that prompt sensitivity is itself a property worth quantifying as part of LLM evaluation.”
From Interaction Topology to Realized Context: Diagnosing Exposure in LLM-Agent Social Simulation
Xu, Y.-W., Han, X.-L., & Ma, C.-H. (2026) · Proceedings of the ACM Turing Award Celebration Conference - China 2026 · doi:10.1145/3833142.3833202
Toward fairer large language models: a divergence-reflection-convergence framework for mitigating cognitive bias
Liu, W., & Zhang, Y. (2026) · Internet Research · doi:10.1108/intr-10-2025-1717
Composable Prompt Modularization in Text-to-SQL Agents: A Practical Approach
Gonçalves, D. C., Neves, B. S., Ferrufino, P. I. V., Costa, D. U. C., Venceslau, A., Franco, W., de Macêdo, J. A. F., & Filho, R. H. (2026) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2026.249296
“The SQL Prompt sequence (Hea-der, Instruction, Schema, and Footer) extends this process to the generation of structured SQL queries, ensuring semantic fidelity and syntactic adherence to the relational schema [Zhuo et al. 2024, Schnabel and Neville 2024, Liu et al. 2025].”
Influential
Using Artificial Intelligence to Access Reproductive Healthcare Information by U.S. States Following Sexual Assault
Stappenbeck, C. A., Westemeier, O., Williams, S., Patton, E. E., Zong, Z. Y., Ross, B. V., & Peddle, A. K. (2026) · Archives of Sexual Behavior · doi:10.1007/s10508-026-03608-w
Verifiable and Reproducible LLM-Based Extraction of AAS Constraints from IDTA Submodel Template PDFs
Jannatabadi, M. H. K., Heppner, S., Miny, T., & Kleinert, T. (2026) · IEEE International Conference on Emerging Technologies and Factory Automation · doi:10.1109/etfa68377.2026.11725266
“Prompt sensitivity has been quantified empirically: small wording changes can produce substantially different extracted records [14].”
Effect of hierarchical prompt engineering on convolutional neural networks generated by large language models for helmet wearing compliance classification
Álava, J. A. P., & Rodas‐Silva, J. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02501-8
“This provides direct empirical support for hypothesis 𝐻𝐻 3 and operationalizes the model-specific prompt-sensitivity phenomenon characterized in Sclar et al. [29]; Zhuo et al. [37].”
Influential

Authors: Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K.
Affiliations: Jilin University (CN)
Shanghai Artificial Intelligence Laboratory (CN)
Chinese University of Hong Kong (HK)
ORCID: Zhang, Songyang
Fang, Xinyu
Lin, Dahua
Chen, Kai
Venue: arXiv
Publisher: arXiv
Published: 2024
Pages: 1950-1976
DOI: 10.48550/arxiv.2410.12405
arXiv: 2410.12405
OpenAlex: W4404782660
Semantic Scholar: 04fd862b74f7161043e7b93e4fa1ca5acb484a06
Cited by: 78
References: 24
Citations by year: 2026: 54 · 2025: 24
FWCI: 22.52
Type: conference paper
Language: en
Open access: bronze
doi.org
Licence: creativecommons.org/licenses/by/4.0/legalcode
Locations: Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion)
arXiv (Cornell University) (submittedVersion, cc-by)
arXiv (Cornell University) (cc-by)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords: prompt sensitivity, large language models, prompt robustness, subjective evaluation, few-shot prompting, model size, reasoning tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks