In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT

“Language models need to identify question tenses to provide correct answers [52, 55].”

“However, it is worth noting that building a high-quality training dataset of such magnitude, i.e., approximately 300B words [1, 18], can pose significant challenges regarding computational resources and associated costs.”

“We then utilize a priori coding, a common deductive approach in HCI, psychology, and usable security that categorize data samples with the guide of established taxonomies or hypotheses [18, 28, 32, 46].”

“We employ 10 widely used benchmark QA datasets in our study, including BoolQ [24], OpenbookQA (OQA) [51], RACE [46], ARC [25], CommonsenseQA (CQA) [67], SQuAD1 [60], SQuAD2 [59], NarrativeQA (NQA) [42], ELI5 [28], and TruthfulQA (TQA) [50].”

“• OpenbookQA (OQA) [51] is a multiple-choice reasoning dataset.”

“For instance, in the CQA dataset, GPT-4 achieves a correctness rate of 79.03%, demonstrating a 2.43% increase compared to GPT-3.”

“The questions are derived from knowledge encoded in ConceptNet [64].”

“The benign roles include an assistant (Assistant), an expert (Expert), an expert using zero-shot chain-of-thought prompt [44] (Expert-CoT), and an expert intended to refuse unanswer-able questions (Expert-R).”

“To achieve this, we leverage BERTopic [32] to automatically topic modeling questions and then apply deductive analysis to assign these topics into broad domains.”

“However, once we modify “assists” to “assist” (TextFooler attack), ChatGPT goes for (D) conquer.”

“5 architecture [23].”

“• RACE [46] is a multiple-choice reading comprehension dataset.”

“Based on the answer types, QA tasks can be generally categorized into four types [41], i.e., yes/no [24], multiple-choice [25, 46, 51, 67], extractive [59,60], and abstractive tasks [28,42,50] (see Table 1 for details).”

“Given questions (and the context if any), QA tasks evaluate a model’s capability in reading comprehension [24, 59, 60], information retrieval [36], logical reasoning [74], and knowledge base [73].”

“To measure this capability, following previous work [43], we calculate the accuracy for YN and MC tasks; the F1 and RougeL metrics for EX and AB tasks, respectively.”

“Previous studies have also shown that LLM is vulnerable to various types of attacks, such as adversarial attacks [27, 30, 37, 39], back-door attacks [11, 21], prompt injection [31, 57], obfuscation [40], and data extraction attacks [19].”

“While large language models (LLMs) have emerged as the foundation for almost all major language tasks, researchers have expressed concerns regarding their capability, robustness, consistency, and potential risks [12, 14, 16, 38, 49, 59, 69, 71].”

“Specifically, VIPER achieves a perplexity score of 304.81 in the BoolQ dataset, while Swap, Nat, Synonyms, TextFooler, and SCPN have perplexity scores of 1286.87, 5936.”

“Following previous work in adversarial attacks [26, 36, 39, 47, 72], the adversary’s objective is to utilize perturbed but semantic-preserving questions to elicit erroneous responses from ChatGPT.”

“• NarrativeQA (NQA) [42] is an abstractive full document comprehension dataset.”

“Given questions (and the context if any), QA tasks evaluate a model’s capability in reading comprehension [23, 58, 59], information retrieval [35], logical reasoning [71], and knowledge base [70].”

“52 architecture [22] and fine-tuned with Reinforcement Learning from Human Feedback (RLHF) [64] to reduce its harmful and untruthful outputs.”

“Swap [13] introduces artificial noises into questions by swapping letters among the words.”

“Based on the answer types, QA tasks can be generally categorized into four types [41], i.e., yes/no [24], multiple-choice [25, 46, 51, 67], extractive [59,60], and abstractive tasks [28,42,50] (see Table 1 for details).”

“We suspect this behavior is possibly due to its reasoning limitations, as it can only generate responses based on training data it has processed [16].”

“While large language models (LLMs) have emerged as the foundation for almost all major language tasks, researchers have expressed concerns regarding their reliability, trustworthiness, robustness, and potential risks [13, 15, 17, 37, 48, 58, 67, 69].”

“While large language models (LLMs) have emerged as the foundation for almost all major language tasks, researchers have expressed concerns regarding their capability, robustness, consistency, and potential risks [12, 14, 16, 38, 49, 59, 69, 71].”

“The questions are derived from science exams (student level ranging from 3rd grade to 9th) that are incorrectly answered by retrieval-based and word co-occurrence algorithms [25].”

“A.1 Detailed Explanation of Datasets • BoolQ [24] is a yes/no reading comprehension dataset.”

“We then utilize a priori coding, a common deductive approach in HCI, psychology, and usable security that categorize data samples with the guide of established taxonomies or hypotheses [17, 29, 33, 47].”

“While Chat-GPT instructed within the system roles has been increasingly used [2, 10, 61] and integrated into various applications [5,6,9], a systematic investigation of the effect of these system roles is still lacking.”

“1 Recent research has shown that ChatGPT obtains capability on par with existing large language models in traditional NLP tasks, such as machine translation, sentiment analysis, and textual entailment [13, 38, 73], and emerging tasks, including code generation and task automation [1,62].”

“• ELI5 [28] is an abstractive full document comprehension dataset.”

“LLMs can be also misused for phishing [54], plagiarism [35, 66], misinformation generation [16], malicious code generation [56], and so on.”

“Previous studies have also shown that LLM is vulnerable to various types of attacks, such as adversarial attacks [26, 29, 36, 39], backdoor attacks [12, 21], prompt injection [30, 56], obfuscation [40], and data extraction attacks [19].”

“Given questions (and the context if any), QA tasks evaluate a model’s capability in reading comprehension [23, 58, 59], information retrieval [35], logical reasoning [71], and knowledge base [70].”

“We refer the audience to [63] for the overview of QA techniques and datasets.”

“1 Recent research has shown that ChatGPT obtains capability on par with existing large language models in traditional NLP tasks, such as machine translation, sentiment analysis, and textual entailment [13, 38, 73], and emerging tasks, including code generation and task automation [1,62].”

“LLMs can be also misused for phishing [53], plagiarism [34, 65], misinformation generation [17], malicious code generation [55], and so on.”

“Question-Answering (QA) is one of the main tasks in NLP [26, 70].”

“Specifically, VIPER achieves a perplexity score of 304.81 in the BoolQ dataset, while Swap, Nat, Synonyms, TextFooler, and SCPN have perplexity scores of 1286.87, 5936.”

“Benefiting from ChatGPT’s instruction-following nature [40], we observe ChatGPT’s response in most cases follow the guide we defined in the prompt, facilitating automatic answer extraction for different QA tasks.”

“Previous studies have also shown that LLM is vulnerable to various types of attacks, such as adversarial attacks [27, 30, 37, 39], back-door attacks [11, 21], prompt injection [31, 57], obfuscation [40], and data extraction attacks [19].”

“LLMs can be also misused for phishing [54], plagiarism [35, 66], misinformation generation [16], malicious code generation [56], and so on.”

“Concretely, we measure the fluency of the generated questions and answers using the perplexity (PPL) metric [58, 75].”

“It can produce hallucination facts [37], exhibit social stereotypes [43], and struggle with mathematical and coding tasks [17], suggesting the potential unreliability.”

“While large language models (LLMs) have emerged as the foundation for almost all major language tasks, researchers have expressed concerns regarding their reliability, trustworthiness, robustness, and potential risks [13, 15, 17, 37, 48, 58, 67, 69].”

“Language models need to identify question tenses to provide correct answers [51,54].”

“We then utilize a priori coding, a common deductive approach in HCI, psychology, and usable security that categorize data samples with the guide of established taxonomies or hypotheses [18, 28, 32, 46].”

“Previous studies have also shown that LLM is vulnerable to various types of attacks, such as adversarial attacks [27, 30, 37, 39], back-door attacks [11, 21], prompt injection [31, 57], obfuscation [40], and data extraction attacks [19].”

“We consider five decision-based adversarial attacks: VIPER [27], Nat [13], Swap [13], Synonyms [15], and SCPN [37].”

“Bagdasaryan and Shmatikov [11] investigate meta-backdoor attacks that cause the language model to generate incorrect outputs with the trigger.”

“We refer to the Library of Congress Classification [20] as our taxon-3 omy as well as initial codes.”

“Previous studies have also shown that LLM is vulnerable to various types of attacks, such as adversarial attacks [27, 30, 37, 39], back-door attacks [11, 21], prompt injection [31, 57], obfuscation [40], and data extraction attacks [19].”

“In addition to these functional concerns, studies analyzing ChatGPT’s characteristics find that it holds pro-environmental and left-libertarian political ideology [34], shows social stereotypes and unfair discrimination [43], and can be easily misled by the wrong knowledge passed in the prompt [76].”

“Question-Answering (QA) is one of the main tasks in NLP [26, 70].”

“[60] evaluate the effect of different prompts on LLMs and find jailbreak prompts decrease LLMs’ reliability in question-answer tasks.”

“For example, [225] shows that ChatGPT significantly under-performs in law and science questions.”

“LLMs as well as LMs for other modalities remain vulnerable to out-of-distribution and adversarial examples [158], [161].”

“…on this front mainly tests models in greater width by expanding the test set with (logic-guided) perturbations and out-of-domain (OOD) examples (Shen et al., 2023; Wang et al., 2023b; Sanyal et al., 2022; Yuan et al., 2023), we explore an orthogonal direction on testing model reasoning in…”

“Jailbreak attacks [21,40,43,75,81,82] are one of the most popular attacks that aim at bypassing the safeguards of LLMs.”

“Finally, Shen et al. (2023a) evaluate the effectiveness of jailbreak prompts against three safeguarding approaches: OpenAI’s Moderation endpoint,23 OpenChatKit Moderation Model,24 and NeMoGuardrails.25 The experiments reveal that all three methods fail to mitigate the jailbreak effectiveness and…”

“…(QA) , while ChatGPT shows potential (Zheng et al., 2023; Shen et al., 2023; Omar et al., 2023), it falters on complex open-domain questions and tasks requiring reasoning (Zheng et al., 2023; Tan et al., 2023) and is vulnerable to adversarial examples and perturbations (Shen et al., 2023).”

“, 2018), capable of fooling even the most advanced models with imperceptible perturbations (Shen et al., 2023).”

“Jailbreak attacks [21,40,43,75,81,82] are one of the most popular attacks that aim at bypassing the safeguards of LLMs.”

“benchmarks across eight knowledge domains, including history, law, and recreation [57].”

“…performance, LLMs are frequently criticized for their limited ability to handle factual information accurately and their tendency to generate hallucinations, especially when faced with questions requiring domain-specific or professional knowledge not covered in their training corpus [2; 48].”

“Viewing this from another perspective, such reverse-engineered prompts could be advantageous, serving as a stepping stone for auxiliary tasks like detecting fake text [12, 13, 21] and generating jailbreak prompts [6,16,19,34].”

“Shen et al. (2023b) showed how jailbreak prompts can undermine LLM reliability in question-answering tasks.”

“Early jailbreak techniques on LLMs involved manual refinement of prompts through trial-and-error, exploiting the randomness of multiple attempts [23, 2].”

“Li et al. [23] and Shen et al. [24] presented the jailbreak method to well-known LLMs by giving specific roles to them inducing privacy and security leaking.”

“Shen et al. [46], and Li et al. [56] exploit indirect prompts to induce LaMDA to elicit erroneous responses.”

“Some prior studies discussed the reliability and potential risks of LLMs, including the robustness to various prompts [9], [14].”

“Shen et al. [54] conducted a comprehensive measurement of ChatGPT’s reliability in generic question-and-answer scenarios, and revealed that ChatGPT often presented opinions as facts or included imaginary specifics without proper qualification.”

“Despite the extensive training and continuous advancement in AI-driven content generation, the potential for errors, misperceptions, and even hallucinatory behavior persists within these applications (Jang & Lukasiewicz 2023, Shen et al. 2023).”

“Accuracy can vary significantly based on the topic and the complexity of the questions [28].”

“The popular integration of ChatBots in our daily life has caught the attention of research communities to assess the performance of these models on various tasks such as providing factual answers [30], automatizing text annotations tasks [20], or assessing the risks of enabling the mass production of toxic content [21…”

“Problems that reduce the accuracy of ChatGPT are false information, biases and hallucinations [39].”

“This issue is especially pressing in high-stakes scenarios such as medical diagnosis (Fox, 1980; Simpkin & Schwartzstein, 2016; Shen et al., 2023), where incorrect predictions can have severe consequences.”

“One of the major issues is the generation of false information [20], [80] and artificial hallucinations [9], which can 1545-5971 © 2024 IEEE.”

“ChatGPT also provides a unique feature, sys-tem messages, which can be used as part of the prompt to guide the model’s behavior (Shen et al., 2023), such as “You are a helpful assistant that can classify sentences as either causal or correlational research findings” which specifies the model to…”

“Additionally, structured tests have been conducted to explore how different characters would respond to the same questions in a role-playing context [3].”

“Early attacks often manually design prompts through trial and error to jailbreak LLMs [23, 42], whose construction needs a lot of experience, and performance is unstable across different LLMs.”

“…generated the data in the first place deserve acknowledgment and fair compensation (copyright law), 2) Controlling data at source follows the design paradigm of prevention, rather than remedial action, and, 3) It reduces the risk of teaching AIs to “game” safety and alignment benchmarks [71], [72].”

“The first study, conducted by [44], aimed to measure the reliability of ChatGPT in a general question-answering scenario.”

“Recent studies have explored the quality, reliability, and security of code generated by LLMs [21], [22], [23], [24], [25].”

“Access to external databases or the internet allows RAG systems to utilize search engines for accurate, real-time information, enhancing reliability with domain-specific expertise [15],[16] and continuous knowledge updates[17].”

“However, LLMs have been shown to be highly sensitive to input prompts [11], [12].”

“Large language models often contain outdated or incomplete information, as training data lack real-time updates [31] and domain-specific expertise [32,33].”

“Finally, Shen et al. (2023) present a first large-scale measurement of ChatGPT’s reliability in a generic question-answering scenario with a set of 5695 questions across ten datasets and eight domains.”

“Recent studies try to automate most of the testing to be able to evaluate ChatGPT performance over thousands or tens of thousands of samples from different publicly available datasets, for example in different types of questions and answer tasks [7, 8].”

“This issue, called “hallucination” [59], [60], [61], refers to generating text that is nonsensical or strays from the original content.”

“5/4 [52], Bard [1], LLaMA-1/2 [64], and PaLM [14] have revolutionized Natural Language Processing (NLP), fostering extensive research on diverse aspects such as fine-tuning [26, 30, 46], alignment [53, 69], reliability [21, 61], and safety [25, 49, 60, 80].”

“We further categorize prompts/responses into specific domains, following the approach of previous works [10,20].”

“This issue, called “hallucination” [61], [62], [63], refers to generating text that is nonsensical or strays from the original content.”

“The reasons are two-fold: (1) it is difficult to assign a reference text to a tuned query; (2) for a reference-free metric, such as perplexity [73,78], the meta classifier struggles to converge during training, as the generators only output placeholder tokens for this nonsensical query in the…”

“Such prompts can be constructed manually [59], through obfuscation (e.g., encoding [68], translation into minority languages [70], or prompt compositionality [68]), or generated automatically using algorithms such as genetic search [71] or tree-of-thought reasoning [7, 35].”

“The adversary conducts jailbreak attacks [10, 19, 43, 50, 52, 98, 114, 116] by using different methods to bypass the safe-guards of LLMs and induce LLMs to complete the given harmful tasks.”

“As LLMs are increasingly deployed through cloud-based ML-as-a-Service (MLaaS) platforms, minimizing inference cost has become critical for both service providers and end-users—enhancing service availability and reducing token-based billing costs.”

“For example, current generative AI, such as openAI ChatGPT, cannot be considered trustworthy since it may provide inaccurate information due to data limitations in law and science (Shen et al., 2023), despite its largely accurate results for clinical decision-making (Johnson et al., 2023).”

“However, large language models (LLMs), epitomized by ChatGPT, have exposed issues such as hallucination, lag in knowledge updates, and the lack of specific domain expertise in their applications [3][4].”

“Conversely, Shen et al. (2023) found that ChatGPT demonstrated strong performance in science and law exam questions.”

“As the focus of this work is on a QA task, we limit the scope and leave out of consideration other studies focusing, e.g., on robustness and reliability checks of LLM code generation [12, 19].”

“5 and ChatGPT-4 models, ChatGPT-4 shows superiority by %2.43 higher accuracy level (Shen et al. , 2023) and with slightly better performance variability across multimodal exam questions in medical education (Gill et al. , 2024).”

“Shen et al. 8 specifically measured the reliability and accuracy of ChatGPT’s output through a quantitative evaluation to determine where ChatGPT’s response is consistently accurate/inaccurate.”

“The emergence of LLMs brought more advanced reasoning capabilities, but they often produce errors when handling tasks beyond their knowledge base [28], [29].”

“Large language models (LLMs) are a particularly notable advancement, showcasing human-like abilities in tasks such as question-answering and object recognition [13, 42, 44, 59].”

“A study by Shen et al. (2023) showed that the of ChatGPT answers varies greatly depending on the question domain, and even small changes in input can produce striking differences in answers.”

“1) Confidence Degrees: Shen et al. [14] induce ChatGPT to output consecutive class probabilities using leakage prompts.”

“LLMs like ChatGPT have demonstrated fascinating and impressive performance in various real-world applications such as online chatting services [2].”

“2023), knowledge out-dating (He, Zhang, and Roth 2022), and the lack of domain-specific expertise (Li et al. 2023c; Shen et al. 2023).”

“Retrieval-augmented generation (RAG), as a technical paradigm that integrates information retrieval with generative models, effectively mitigates inherent limitations of large language models (LLMs), such as data freshness [1], hallucinations [2–4], and the lack of domain-specific knowledge [5,6].”

“reasoning, symbols manipulation, and code generation (Chen et al., 2019; Kojima et al., 2022; Cohn and Hernandez-Orallo, 2023; Frieder et al., 2023; Ray, 2023; Rozière et al., 2023; Shen et al., 2023).”