Benchmarking Cognitive Biases in Large Language Models as Evaluators

“Large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022) adapted to follow various kinds of instructions have been popularly utilized for several natural language tasks.”

“We cover a wider demographic of current popular language models and an overall different focus on QA as opposed to other domains such as math and reason.”

“We additionally ask each evaluator to analyze generation quality along several different aspects such as “coherence, accuracy, factu-ality, and helpfulness” following (Bai et al., 2023) and (Zheng et al., 2023).”

“…research works have investigated their utility in various downstream tasks, such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual…”

“Here, we computed the RBOs between each individual annotator and machine preferences and averaged them.”

“Large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022) adapted to follow various kinds of instructions have been popularly utilized for several natural language tasks.”

“The general standard for testing a model’s capabilities is benchmarking its performance on static evaluation suites such as Fan et al. (2019) and Wang et al. (2020).”

“We calculated the Rank-Biased Overlap (RBO) score (Webber et al., 2010) to measure the agreement between human preferences and model evaluations in ranking-generated texts across 16 different LLMs. RBO, which can vary from 0 (non-conjoint) to 1 (identical), assigns more weight to the top k items in…”

“(Ross and Sicoly, 1979) is a cognitive bias that refers to the tendency to have a higher opinion of oneself or to more easily accept ideas if they match one’s own.”

“As we are looking to test the ability of language models to perform as unbiased evaluators to judge response quality and correctness, the Q/A response format presents the most natural setting for these comparisons.”

“Order Bias is an evaluation bias we observe when a model tends to favor the model based on the order of the responses rather than their content quality.”

“BigBench (Srivastava et al., 2023) is a collection of benchmarks that look to probe the abilities of language models over a diverse range of tasks.”

“Some recent works (Zhao et al., 2021; Liu et al., 2022; Lu et al., 2022) have shown that the order of training examples in GPT-3 could lead to differences in accuracy between near chance and near state-of-the-art.”

“H ARNESS (Gao et al., 2021), MMLU (Hendrycks et al., 2021), HELM (Liang et al., 2022) and BIG-Bias B ENCH (Srivastava et al., 2023), only focus on general LLM performance but do not explore their capabilities as evaluators.”

“Here, we computed the RBOs between each individual annotator and machine preferences and averaged them.”

“We collect a set of 50 question-answering instructions from two well-established benchmarking datasets: B IG B ENCH (Srivastava et al., 2023) and E LI 5 (Fan et al., 2019).”

“H ARNESS (Gao et al., 2021), MMLU (Hendrycks et al., 2021), HELM (Liang et al., 2022) and BIG-Bias B ENCH (Srivastava et al., 2023), only focus on general LLM performance but do not explore their capabilities as evaluators.”

“We specifically utilize the strategyQA (Geva et al., 2021) dataset, which was constructed by crowd-sourcing questions from writers as well as their responses with short justifications.”

“(Xu et al., 2023b), K OALA (Geng et al., 2023), W IZARD LM (Xu et al., 2023a), MPT (Team, 2023)”

“Order Bias is an evaluation bias we observe when a model tends to favor the model based on the order of the responses rather than their content quality.”

“In addition to the original instruction, we follow a similar setup from (Shi et al., 2023) where we include irrelevant information about one of the comparand models to test the ability of evaluators.”

“(Butts et al., 2019; Västfjäll et al., 2014) is a cognitive bias that denotes a decrease in empathy as the number of identifiable individuals increases.”

“…such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023; Gekhman et al., 2023; Luo et al., 2023), and more.”

“…such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023; Gekhman et al., 2023; Luo et al., 2023), and more.”

“…such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023; Gekhman et al., 2023; Luo et al., 2023), and more.”

“…such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023; Gekhman et al., 2023; Luo et al., 2023), and more.”

“…such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023; Gekhman et al., 2023; Luo et al., 2023), and more.”

“(Schenk, 2010; Zheng et al., 2023) The evaluator tends to favor responses that are either shorter or longer in length.”

“Order bias has been extensively studied (Jung et al., 2019; Wang et al., 2023a; Zheng et al., 2023), and it is well-known that language models can be influenced by the ordering of the responses in their evaluations.”

“…investigated their utility in various downstream tasks, such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al.,…”

“While biases have been well-known to exist in LLMs (Wang et al., 2023b; Talboy and Fuller, 2023; Wu and Aji, 2023), many recent works investigating the behaviors of LLMs have also uncovered similarities with cognitive biases.”

“Along this aspect, while our work is close to Wu and Aji (2023), who investigate biases related to fabricated factual and grammatical errors, our work is much more comprehensive in terms of the number of LLMs analyzed, the types of biases analyzed and the creation of an open benchmark.”

“Our work in this direction overlaps directly with Bai et al. (2023) and Zheng et al. (2023), who propose a Language-Model-as-an-Examiner benchmark and LLM-as-a-judge to study the capability of LLMs to emulate human preferences.”

“…downstream tasks, such as machine translation (Kocmi and Feder-mann, 2023), summarization (Shen et al., 2023; Gao et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023; Gekhman et al., 2023; Luo et…”

“For the pairwise human bias experiments, we referenced the interface design from Hayati et al. (2021).”

“Order bias has been extensively studied (Jung et al., 2019; Wang et al., 2023a; Zheng et al., 2023), and it is well-known that language models can be influenced by the ordering of the responses in their evaluations.”

“The online forum consists of a community for individuals to ask various questions, and answers are provided in a format that is comprehensible to five-year-olds, along with assigned scores based on community votes.”

“Along this aspect, while our work is close to Wu and Aji (2023), who investigate biases related to fabricated factual and grammatical errors, our work is much more comprehensive in terms of the number of LLMs analyzed, the types of biases analyzed and the creation of an open benchmark.”

“We choose the strategyQA dataset to generate responses that require multi-step reasoning to effectively benchmark the ability of models to comprehend and compare the quality between two different explanations.”

“This may be due to the increased complexity of the task (Dziri et al., 2023) where the ranking of N generations may become much more difficult as N gets larger (since the task complexity increases).”

“We cover a wider demographic of current popular language models and an overall different focus on QA as opposed to other domains such as math and reason.”

“LLMshavealsobeennotedforegocentricbiaswhichcould affectevaluationsasmoreandmoreLLM-generatedtextappearsinsource texts 63 .”

“…et al., 2023; Zeng et al., 2023; Yuan et al., 2023; Fu et al., 2023; Li et al., 2024), self-refinement (Saunders et al., 2022; Madaan et al., 2023; Lee et al., 2023; Shridhar et al., 2023), and constitutional AI (Bai et al., 2022), LLMs are increasingly used to provide assessment, supervision,…”

“Park et al. conducts a comparative evaluation of LLMs for automatically extracting knowledge from scientific literature to understand protein interactions and pathway knowledge Park et al. [2023].”

“These benchmarks vary in their emphasis on different aspects of content risk, covering themes such as toxicity [29, 30], bias [31, 32], privacy [33], and misinformation [34, 35].”

“Koo Biased Overlap (RBO) between model outputs and human bias assessments was only 49.6%, revealing significant inconsistency 50 .”

“As a result, when the system attempts to balance topic distribution, it may reject high-quality, genuine submissions in that area simply because the topic appears over-represented, thereby squeezing out legitimate competition 140 .”

“Furthermore, several other benchmarks for assessing bias in LLMs have been proposed for specific types of bias, including cognitive [20], gender-occupational [19], religion [12], and racial [30].”

“As an additional non-LLM baseline on the privacy axis, a regex + ontology rule-based judge achieved 81% accuracy (162/200) against the human labels, closely tracking the LLM judge and confirming that the high privacy-leakage rate is detectable without an LLM adjudicator.”

“This tendency suggests a preference for brevity, with the model tending toward shorter responses than human responses [43], [44].”

“Despite its promising potential, recent studies have highlighted the vulnerabilities and biases of LLM-as-a-Judge [79, 72, 28, 4, 79, 21, 59, 54].”

“More recently, GPT-4 and other high-end LLMs were used to assess answer quality of other models, but this strategy has its pitfalls and downsides [22].”

“To further reduce time complexity and avoid the curse of dimensionality [52], we apply Principal Component Analysis (PCA) for dimension reduction.”

“To find the area of the triangle, we can use the formula for the area of a triangle with coordinates A(x1, y1), B(x2, y2), and C(x3, y3): Area = (1/2) * |(x1(y2 - y3) + x2(y3 - y1) + x3(y1 - y2))| In this case, the coordinates are A(0, 0), B(-1, 1), and C(3, 3).”

“Although there has been an increasing amount of attention to using LLMs, especially GPT-4, to evaluate LLM output, such a paradigm can lead to overemphasis on stylistic or surface-level similarities to ground truth, while missing deeper biases that affect fairness [61, 97, 100].”

“Positional bias, response-length bias, and self-preference bias are well-known biases that could significantly impact the quality and objectivity of the LLM-as-judge benchmarks [24], [30], [31], [32], [33].”

“…often aligns well with expert judgments [32], judge models from similar architectural families may share systematic biases and over rely on surface-level fluency rather than factual correctness, highlighting the continued need for complementary human judgment and retrieval based verification [33].”

“This tendency, referred to as self-preference bias [11], [12], [13], occurs when a model rates its own generated output more favorably than work produced by other systems.”

“Koo et al. (2024) systematically measured six categories of cognitive bias in LLM evaluation and showed that, across 16 tested models, approximately 40% of pairwise comparisons displayed biased tendencies, while human-model preference agreement was only 44%.”

“They also exhibit choice-order bias (Dominguez-Olmedo et al., 2023; Wang et al., 2023; Shi et al., 2024) as well as a variety of other biases (Koo et al., 2023).”

“Within the first direction, Koo et al. [2023] and Liu et al. [2023b] examine the frequency at which an LLM assigns higher scores to their own outputs over others’.”

“Broader cognitive biases tied to training distributions further compound these issues [27, 31].”

“A persistent finding line documents systematic biases in LLM judges: positional bias, verbosity bias, and self-preference effects [Zheng et al., 2023, Koo et al., 2024, Panickssery et al., 2024].”

“However, data-driven LLMs inevitably introduce new biases in the outputs generated from evaluation prompts (Wu and Aji, 2023; Zheng et al. 2024; Koo et al. 2024; Chen et al. 2024b; Stureborg, Alikaniotis, and Suhara, 2024, Wang et al. 2024), with remedies such as chain-of-thought and fine-tuning…”

“…[4] demonstrated LLMs’ effectiveness in capturing semantic distinctions essential for inter-disciplinary research communication, while Chow, Kao, and Li [5] showed that while LLMs can automate subject heading generation, human expertise remains crucial for understanding nuanced contextual elements.”

“Previous studies [34], [35], [36], [37] have indicated that LLMs are not well-suited for accurate ranking, particularly in scoring and direct comparisons.”

“For example, typical cases of this can be seen in the underestimation/distortion of the order of events [46], [73] or in the omission of details considered irrelevant [73], [74].”

“In the non-feedforward condition, the AI’s self-evaluation comments tended to display conviction and achievement without meaningful fluctuations, reflecting the typical overconfidence bias reported in previous studies [7], [8].”

“Recent research [11, 29] further argues that LLMs not only mimic human-like behaviour but also express cognitive biases themselves.”

“Just as LLMs have been shown to display cognitive biases in generation tasks [23, 51, 53, 70], they also exhibit cognitive biases when employed as judges [36].”

“Cognitive-bias work in LLMs [2], [15], [16], [17] probes biases inside models on medical or general benchmarks, not bias expressed in corporate disclosures, and stops at detection without mitigation.”

“Such discrepancies are especially problematic in cybersecurity settings, where false negatives can directly translate into undetected harmful outputs, undermining trust in automated evaluation pipelines (Koo et al. 2023; Liu et al. 2023; Stureborg et al. 2024; Eiras et al. 2025).”

“…work documents several sources of judge unreliability, including prompt sensitivity, distribution shift, prompt complexity, and leniency [4], [5], response verbosity, position bias, cognitive bias, and self-preference [23], [24], [25], and over-refusal as a complementary failure mode [26], [27].”

“Model evaluators can show bias, vulnerability to evaluation format, or confusion between quality criteria, which means that agreement metrics alone may not reveal why a judgement differs from the human reference [34,35].”

“That is, Follower nodes tend to blindly conform to the views of the initiating or dominant node rather than adhering to objective facts [17], [18].”

“In response to these limitations, LLM-as-judge methodologies have emerged as scalable evaluation alternatives, but they introduce their own concerns regarding evaluator bias, criterion confusion, and expert misalignment [7, 18, 33].”

“To reduce self-preference bias [30], we use three judges from unrelated training lineages, GLM4:9B (Zhipu), Granite3.”

“…interaction-level bias , includes positional (Wang et al., 2024a) and verbosity biases (Koo et al., 2024; Saito et al., 2023), as well as sycophancy (Zheng et al., 2023), persuasive responses (Stengel-Eskin et al., 2025), authority bias (Ye et al., 2024) and bandwagon effects (Koo et al., 2024).”