No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

Accurately judging response quality in high-stakes domains hinges on identifying whether a response is correct, which can require complex reasoning and domain knowledge.

“In these settings, accurately judging response quality hinges on identifying whether a response is correct, which itself can require complex reasoning and domain knowledge (Tan et al., 2025;Zheng et al., 2023).”

• cites Tan et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zheng et al., 2023 sourcePartly supported

  Section 3.3 (p. 6) says LLM judges can fail to grade math and reasoning answers because they do not know the correct answers, supporting the link between judging quality, correctness, and reasoning. The paper does not establish that claim for high-stakes domains or specifically discuss the domain knowledge such settings may require. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

In high-stakes domains such as business and finance, correctness outweighs other attributes.

“In some high-stakes domains such as business and finance, however, correctness outweighs other attributes (Bommasani et al., 2022;Feuer et al., 2025;Krumdick et al., 2024).”

• cites Bommasani et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Feuer et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Krumdick et al., 2024 sourcePartly supported

  The Conclusion says numerical reasoning is critically important in business and finance because small errors can incur large costs. That supports the importance of correctness in these high-stakes domains, but the paper does not say correctness outweighs other attributes. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Self-preference bias is a common judgment bias in which a model tends to overrate its own responses.

“One common judgment bias is self-preference (Panickssery et al., 2024), in which a model tends to overrate its own responses.”

• cites Panickssery et al., 2024 sourceSupported

Prior studies have demonstrated that LLMs can reach high agreement with human annotators when grading correctness of responses to short-answer questions on existing datasets.

“Studies (Thakur et al., 2025;Badshah & Sajjad, 2024;Henkel et al., 2024;Badshah et al., 2025) have demonstrated that LLMs can reach high agreement with human annotators when grading the correctness of responses to short-answer questions on existing datasets.”

• cites Thakur et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Badshah & Sajjad, 2024 sourceSupported
• cites Henkel et al., 2024 sourcePartly supported

  The Introduction reports that GPT-4 achieved close to human-level agreement when classifying short-answer responses as correct or incorrect (κ = 0.70 versus 0.75). However, §2.1 says the authors created a new dataset from Carousel student responses, so this paper does not support the claim’s characterization of its evaluation as being on an existing dataset. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Szymanski et al. (2025) found only moderate agreement between GPT-4's ratings and domain professionals on a dataset of 25 questions in dietetics and mental health.

“Szymanski et al. (2025) examines LLM-as-a-judge in the domains of dietetics and mental health by comparing GPT-4's ratings to those of domain professionals, finding only moderate agreement on a dataset of 25 questions.”

• cites Szymanski et al. (2025) sourceSupported

Using a model's own references exacerbates the self-preference bias compared to providing no reference at all.

“Specifically, using a model's own references exacerbates the self-preference bias compared to providing no reference at all.”

When provided with a human reference, even the worst LLM Judge outperforms the best baseline method.

“When provided with a human reference, even the worst LLM Judge outperforms the best baseline method.”

The effect of reference type on judge agreement is robust across both datasets used to create VERDICTS, other models, and LLM Juries.

“demonstrating this effect is robust across both datasets used to create VERDICTS (Section I), other models (Section J) and LLM Juries (Section H).”

LLM-as-a-Judge performance is sensitive to the provided reference.

“LLM-as-a-Judge performance is sensitive to the provided reference.”

The disparity in agreement is most pronounced for benchmark questions the judge model answers incorrectly.

“The disparity in agreement is most pronounced for benchmark questions the judge model answers incorrectly.”

The effect of low false positive rate and high false negative rate is not observed with the Random reference.

“However, this effect is not observed with the Random reference.”

For questions the judge model was unable to correctly answer, there are large differences in agreement between using different reference types.

“For questions the judge model was unable to correctly answer, there are large differences in the agreement between using different reference types.”

• AssumptionRests on an assumption: A judge model's correctness when used as a candidate model indicates its ability to answer the underlying benchmark question.

On VERDICTS, whether an LLM Judge agrees with human experts is strongly related to its ability to answer the underlying benchmark question and the correctness of the provided reference.

“We demonstrate that on VERDICTS, whether an LLM Judge agrees with human experts is strongly related to its ability to answer the underlying benchmark question and the correctness of the provided reference (Figure 1).”

• AssumptionRests on an assumption: Human expert annotations of response correctness are a reliable ground truth for evaluating automated grading methods.
• AssumptionRests on an assumption: A judge model's correctness when used as a candidate model indicates its ability to answer the underlying benchmark question.

Even when provided with a Wrong reference that should make nearly every other response incorrect, LLM Judges were still frequently judging responses to be correct.

“Even when provided with a Wrong reference that should make nearly every other response incorrect, LLM Judges were still frequently judging responses to be correct.”

Providing judges with expert-written references largely mitigates the low agreement on questions the judges cannot answer.

“We demonstrate that providing the judges with expert-written references largely mitigates this issue, highlighting the limits of using LLM-as-a-Judge without any form of human verification.”

LLM Judges are more reliable than other automated grading methods.

“While we observe that LLM Judges are more reliable than other grading methods, our findings reveal a clear pattern in LLM Judge performance”

15 (37.5%) of the provided GPT-4-generated reference answers for the 40 MT-Bench math and reasoning questions were either incorrect or inconsistent.

“we reviewed the provided GPT-4-generated reference answers for these 40 questions and found that 15 (37.5%) were either incorrect or inconsistent.”

• AssumptionRests on an assumption: The MT-Bench math and reasoning subset is evaluated with references by default and leads to responses that tend to be unambiguously correct or incorrect.

Model performance is generally correlated between BFF-Bench and (C)MT-Bench.

“We find model performance to be generally correlated between the two benchmarks.”

When not provided with a correct reference, LLM Judges show high agreement with human experts only on questions the judges were able to correctly answer themselves.

“when not provided with a correct reference, judges show high agreement with human experts only on questions the judges were able to correctly answer themselves.”

• Also stated in: § 5 Results · plain

  “Without a human-written reference, LLM Judges only achieve high agreement on questions they were able to answer as candidates.”

• AssumptionRests on an assumption: Human expert annotations of response correctness are a reliable ground truth for evaluating automated grading methods.
• AssumptionRests on an assumption: A judge model's correctness when used as a candidate model indicates its ability to answer the underlying benchmark question.

There is a significant gap between the Human references and the Wrong references.

“There is also a significant gap between the Human references and the Wrong references.”

Higher agreement with human annotators was observed in pairwise grading than in single grading.

“As expected, we observed higher agreement with the human annotators in the pairwise grading setup than in the single grading setup.”

• AssumptionRests on an assumption: Human expert annotations of response correctness are a reliable ground truth for evaluating automated grading methods.

Without a provided reference, some LLM Judges and reward models perform on par with heuristics.

“However, without the provided reference, some LLM Judges and reward models perform on par with heuristics.”

For every reference type, on average models have a higher false positive rate when grading their own responses.

“For every reference type, we see that on average models have a higher FPR when grading their own responses.”

VERDICTS contains expert judgments of correctness for 1,200 LLM responses.

“a dataset of expert judgments of correctness for 1,200 LLM responses to evaluate the reliability of methods of pairwise and single grading.”

On questions the judge had answered correctly, providing human references still improves performance.

“On questions the judge had answered correctly, providing human references still improves performance.”

We find no meaningful variation in single grading results when using a separate prompt with rephrased instructions.

“To ensure our results are robust to the underlying prompt, we re-run our single grading experiments with a separate prompt that contains a rephrasing of the same instructions.These results are discussed in Section A.6, but we find no meaningful variation.”

When GPT-4o uses its own responses as a reference, Cohen's κ decreases from 0.86 to 0.16 in the pairwise case on questions GPT-4o could not answer.

“when GPT-4o uses its own responses to questions as a reference (i.e., Self response), the Cohen's κ decreases from 0.86 to 0.16 in the pairwise case on questions GPT-4o could not answer.”

The GPT-4o judge's performance does not shift significantly with the addition of chain-of-thought.

“We do not observe significant shifts in performance with the addition of chain-of-thought.”

Across both source datasets, when the LLM judge is not provided a reference (None or Self), there is a large difference in Cohen's κ between questions the judge answered correctly and questions it answered incorrectly.

“Across both datasets, when the Judge is not provided a reference (None, Self) there is a large difference in κ.”

For every judge model in every setting, agreement with human annotators increases when using domain expert-written references.

“For every judge model in every setting, we observe an increase in agreement with the human annotators when using domain expert-written references.”

The GPT-4o judge's performance is not sensitive to the sampling temperature.

“The LLM Judge performance is not sensitive to the temperature for this model.”

Under Prompt One, Qwen 2.5 7B's overall judge accuracy was 0.24 ± 0.06.

“Qwen 2.5 7B 0.24 ±0.06 0.43 ±0.06 0.63 ±0.05 0.31 ±0.10 0.37 ±0.10 0.54 ±0.09 0.08 ±0.07 0.21 ±0.10 0.57 ±0.08”

Under Prompt One, Llama 3.3 70b's overall judge accuracy was 0.42 ± 0.05.

“Llama 3.3 70b 0.42 ±0.05 0.53 ±0.05 0.78 ±0.04 0.62 ±0.09 0.54 ±0.08 0.74 ±0.07 0.13 ±0.06 0.23 ±0.11 0.69 ±0.10”

Ablating the reasoning level of frontier models has little overall impact on LLM judge performance.

“Additionally, we ablate the reasoning level for these models in Table 13 but find little overall impact.”

The paper's main result is preserved across reasoning levels.

“Our main result is preserved across reasoning levels.”

Higher reasoning effort can narrow the no-reference gap for some models (e.g., GPT-5.4 at high effort).

“Notably, higher reasoning effort can narrow the no-reference gap (e.g., GPT-5.4 at high effort)”

Using LLM-as-a-Judge does not alleviate the need for human oversight.

“Using LLM-as-a-Judge does not alleviate the need for human oversight.”

Verifying responses generated by a model rather than manually writing and checking answers via annotators is sufficient in this setting.

“Verifying responses generated by a model rather than manually writing and checking answers via annotators is sufficient in this setting.”

The verification process still requires some amount of human effort, and this is unavoidable for developing trustworthy LLM-as-a-Judge pipelines.

“Although this verification process still requires some amount of human effort, our results suggest that this is unavoidable for developing trustworthy LLM-as-a-Judge pipelines.”

The most important part of human-written references is their correctness, rather than any other superficial or stylistic attributes.

“This suggests that the most important part of human-written references is their correctness, rather than any other superficial or stylistic attributes.”

The pairwise grading setting represents a simplification of the real-world use-case by removing instances where a model would have to distinguish correctness between two correct or two incorrect responses.

“In our analysis of the pairwise grading task, however, our setting represents a simplification of the real-world use-case by the removal of instances in which a model would have to distinguish correctness between two correct or two incorrect responses.”

Judges are sensitive to the correctness of the reference, and providing a slightly incorrect reference can in some cases be worse than providing a completely unrelated reference or no reference at all.

“This evidence suggests that judges are sensitive to the correctness of the reference, and providing a slightly incorrect reference can in some cases be worse than providing a completely unrelated reference or no reference at all.”

Because of the simplified pairwise setting, pairwise grading results should be interpreted only relatively among judge models rather than absolutely.

“Because of this simplified setting, the pairwise grading results should be interpreted only relatively (among judge models) rather than absolutely.”

The jury does not escape the core limitation identified in the paper.

“The jury does not escape the core limitation we identified.”

Incorrect information contained in a Wrong reference leads judges to erroneously classify correct responses as incorrect.

“which suggests that the incorrect information contained in the reference leads judges to erroneously classify correct responses as incorrect.”

The effect of reference availability on LLM judge agreement is not specific to either of the two source datasets.

“These results demonstrate that the effect is not specific to either of the two source datasets.”

The paper's results are robust across the temperature and chain-of-thought hyperparameters.

“We do not observe a sensitivity to either of these two factors, providing evidence that our results are robust across these two hyperparameters.”

Feuer et al. (2025) find that judge models often ignore response correctness in their quality assessments.

“Feuer et al. (2025) specifically study how well standard LLM-as-a-Judge pipelines capture correctness and find that judge models often ignore response correctness in their quality assessments.”

• cites Feuer et al. (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.