LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment

Prompt engineering is the most typical mechanism for instructing the judge.

“Prompt engineering is the most typical mechanism for instructing the judge.”

Cross-informed voting, where each agent observes others' outputs before voting, achieves stronger agreement with gold-standard rankings than independent voting in de-identification model selection.

“Cross-informed voting, where each agent observes others' outputs before voting, achieves stronger agreement with gold-standard rankings than independent voting in de-identification model selection [148].”

• cites [148]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Several studies examine judge reliability and robustness through inter-judge agreement, consistency, and task-specific downstream metrics such as diagnostic recall or safety detection.

“Beyond predictive performance, several studies examine judge reliability and robustness through inter-judge agreement, consistency, and task-specific downstream metrics such as diagnostic recall or safety detection [132].”

• cites [132]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A small subset of studies (n=6) applies LLM-as-a-Judge to public health and governance, as well as research and evidence synthesis tasks.

“A small subset of studies (n=6) applies LLM-as-a-Judge to public health and governance [77,137,149], as well as research and evidence synthesis tasks [30,43,96].”

• cites [77]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [137]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [149]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Across Clinical Decision Support studies, LLM-as-a-Judge is evaluated with a diverse but coherent set of metrics.

“Across Clinical Decision Support studies, LLM-as-a-Judge is evaluated with a diverse but coherent set of metrics.”

Closed-source models appear to be selected as judges more frequently than open-source models.

“Overall, closed-source models appear to be selected as judges more frequently than open-source models.”

Many studies apply rubric-based or Likert-scale evaluations across clinically relevant dimensions, as in LiveMedBench and CounselBench.

“Many studies further apply rubric-based or Likert-scale evaluations across clinically relevant dimensions, as in LiveMedBench [155] and CounselBench [81].”

• cites [155]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [81]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Mental and behavioral health represents another major area within Clinical Decision Support (n=18, 33.3%).

“Mental and behavioral health represents another major area within Clinical Decision Support (n=18, 33.3%).”

Rubric-based prompting is predominant, encoding structured evaluation criteria with explicit score-level descriptors.

“For prompting strategies, rubric-based prompting is predominant, which encodes structured evaluation criteria with explicit score-level descriptors.”

Clinical NLP studies often embed LLM judges within end-to-end systems for summarization, information extraction, or de-identification, where task-specific downstream metrics (e.g., hallucination detection rate, extraction accuracy, and de-identification precision and recall) are used to evaluate operational performance.

“Clinical NLP studies also often embed LLM judges within end-to-end systems for summarization, information extraction, or de-identification, where task-specific downstream metrics (e.g., hallucination detection rate, extraction accuracy, and de-identification precision and recall) are used to evaluate operational performance [82,97].”

• cites [82]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [97]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Within Clinical Decision Support, most studies focus on diagnosis and screening tasks (n=19, 35.2%).

“Within Clinical Decision Support, most studies focus on diagnosis and screening tasks (n=19, 35.2%), reflecting the use of LLM-as-a-Judge to assess diagnostic accuracy and triage decisions.”

In Clinical NLP tasks (n=28), LLM-as-a-Judge is mainly used for clinical documentation tasks (n=21, 75.0%).

“In Clinical NLP tasks (n=28), LLM-as-a-Judge is mainly used for clinical documentation tasks (n=21, 75.0%), reflecting a strong focus on evaluating generated and summarized clinical notes.”

One evaluation assesses alignment with human experts, commonly using agreement measures such as Cohen's κ, Krippendorff's α, Fleiss' κ, or agreement rate, together with correlation-based metrics such as Pearson's r , Spearman's ρ, and Kendall's τ to measure concordance with clinician ratings.

“One evaluation assesses alignment with human experts, commonly using agreement measures such as Cohen's 𝜅, Krippendorff's 𝛼, Fleiss' 𝜅, or agreement rate, together with correlation-based metrics such as Pearson's 𝑟 , Spearman's 𝜌, and Kendall's 𝜏 to measure concordance with clinician ratings [147].”

• cites [147]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Prompt engineering is used in 132 of 134 studies (98.5%).

“Regarding the judge techniques, prompt engineering is used in 132 of 134 studies (98.5%) (Figure 6b).”

Existing studies evaluate Completeness, which assesses whether all key information is included.

“Existing studies also evaluate Completeness, which assesses whether all key information is included [6,52,110,111]”

• cites [6]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [52]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [110]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [111]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Prior work has used LLM-as-a-Judge to evaluate Presentation, including conciseness, clarity, and readability.

“In addition, prior work has used LLM-as-a-Judge to evaluate Presentation, including conciseness, clarity, and readability”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [67]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [93]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [111]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [145]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Our analysis identifies eight rubric types used in collected studies.

“Our analysis identifies eight rubric types used in collected studies (Table 3a).”

Liu et al. [84] used LLM-as-a-Judge to assess emotional supportiveness and cultural awareness in a dataset designed to support individuals in distress.

“For example, in a dataset designed to support individuals in distress, Liu et al. [84] used LLM-as-a-Judge to assess emotional supportiveness and cultural awareness.”

• cites [84]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

When LLM judges produce discrete labels, studies often report classification metrics, including accuracy, precision, recall, F1-score, and ROC-AUC.

“When LLM judges produce discrete labels, studies often report classification metrics, including accuracy, precision, recall, F1-score, and ROC-AUC [121]”

• cites [121]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Four-class clinical risk grading in MedVAL remains in the moderate range (κ = 0.67).

“for example, four-class clinical risk grading in MedVAL remains in the moderate range (𝜅 = 0.67) [1].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In one study, the same judge that reached 0.83 agreement with at least one expert dropped to 0.51 against majority-of-experts judgments.

“in one study, the same judge that reached 0.83 agreement with at least one expert dropped to 0.51 against majority-of-experts judgments [23]”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Many rubrics evaluate whether an output is factual, complete, safe, readable, and clinically useful.

“Many rubrics evaluate whether an output is factual, complete, safe, readable, and clinically useful (Table 3).”

Yao et al. [158] report 0.80 raw agreement but κ = 0.59 on the same dataset.

“Yao et al. [158] report 0.80 raw agreement but 𝜅 = 0.59 on the same dataset, showing that prevalence-corrected reliability can be substantially lower when one rating category dominates.”

• cites [158]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The reviewed studies show that LLM-as-a-Judge is often used for more than checking whether an answer is correct.

“The reviewed studies show that LLM-as-a-Judge is often used for more than checking whether an answer is correct.”

The highest Cohen's κ value is reported for GPT-4o in medical image quality assessment (0.88).

“The highest value is reported for GPT-4o in medical image quality assessment (𝜅 = 0.88) [88]”

• cites [88]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Zhou et al. [168] use Likert-style reasoning scores aligned with expert ratings.

“For example, Zhou et al. [168] use Likert-style reasoning scores aligned with expert ratings”

• cites [168]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The highest agreement value appears in MedGUIDE, where an ensemble of GPT-4o-mini, Claude-3.5-Haiku, Gemini-2.5-Flash, and DeepSeek-V3 is used to score guideline-grounded multiple-choice questions, with validation on 500 human-reviewed samples.

“the highest value appears in MedGUIDE, where an ensemble of GPT-4o-mini, Claude-3.5-Haiku,Gemini-2.5-Flash,and DeepSeek-V3 is used to score guideline-grounded multiple-choice questions, with validation on 500 human-reviewed samples [79].”

• cites [79]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The lowest Cohen's κ value (0.59) is observed for DeepSeek-V3 evaluating educator dialogues and personalized discharge summaries.

“The lowest value (𝜅 = 0.59) is observed for DeepSeek-V3 evaluating educator dialogues and personalized discharge summaries [158].”

• cites [158]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Mental-health counseling rubrics with more subjective language register 0.73.

“whereas mental-health counseling rubrics with more subjective language register 0.73 [81].”

• cites [81]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Studies quantify alignment using different metrics depending on task structure, output format, and rating scale.

“However, studies quantify this alignment using different metrics depending on task structure, output format, and rating scale.”

Across 13 examined studies reporting Pearson or Spearman correlation, judge-expert correlation ranges from 0.40 to 0.94 (median 0.69, mean 0.68).

“Across 13 examined studies reporting Pearson or Spearman correlation (Figure 10; 10 Pearson, 3 Spearman), judge-expert correlation ranges from 0.40 to 0.94 (median 0.69, mean 0.68).”

Across 13 examined studies reporting agreement against expert judgments, judge-expert concordance ranges from 0.66 to 0.96 (median 0.83, mean 0.83).

“Across 13 examined studies reporting agreement against expert judgments (Figure 8), judge-expert concordance ranges from 0.66 to 0.96 (median 0.83, mean 0.83).”

A key feature of Medical Knowledge & QA is the use of reasoning-oriented rubrics, in which judges assess logical validity, clinical reasoning quality, evidence use, and safety.

“A key feature is the use of reasoning-oriented rubrics, in which judges assess logical validity, clinical reasoning quality, evidence use, and safety.”

Claude-Opus-4.6 performance drops substantially for perturbed concepts (κ = 0.24).

“but its performance drops substantially for perturbed concepts (𝜅 = 0.24) [127].”

• cites [127]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A three-judge ensemble (GPT-4, Claude, DeepSeek) reaches 0.90 on rubric-anchored adequacy and safety items in NSCLC care.

“a three-judge ensemble (GPT-4, Claude, DeepSeek) that reaches 0.90 on rubric-anchored adequacy and safety items in NSCLC care [22]”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

GPT-4o reaches 0.88 in medical image quality assessment.

“GPT-4o reaches 0.88 in medical image quality assessment [88]”

• cites [88]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Several studies quantify correctness using counts or proportions of correct, incorrect, or missing facts, or through structured factuality scores.

“Several studies quantify correctness using counts or proportions of correct, incorrect, or missing facts, or through structured factuality scores [133].”

• cites [133]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Several studies include meta-evaluation and efficiency measures, such as judge-expert correlation, statistical discrimination tests, evaluation cost, runtime, and confidence calibration.

“Several studies further include meta-evaluation and efficiency measures, such as judge-expert correlation, statistical discrimination tests, evaluation cost, runtime, and confidence calibration [10,168].”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [168]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

GPT-4o reaches only r = 0.483 on counseling responses evaluated across five subjective safety dimensions.

“In contrast, GPT-4o reaches only 𝑟 = 0.483 on counseling responses evaluated across five subjective safety dimensions [18].”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

DeepSeek-R1 reaches r = 0.938 on the behavior dimension of a five-dimension psychosocial safety rubric.

“DeepSeek-R1 reaches 𝑟 = 0.938 on the behavior dimension of a five-dimension psychosocial safety rubric [90]”

• cites [90]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Medical QA evaluation emphasizes fine-grained factuality and statement-level error analysis.

“Medical QA evaluation also emphasizes fine-grained factuality and statement-level error analysis.”

GPT-4o-mini for personalized longevity recommendations has a Cohen's κ of 0.87.

“followed by GPT-4o-mini for personalized longevity recommendations (𝜅 = 0.87) [57].”

• cites [57]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Claude-Opus-4.6 shows good alignment with human annotators for unperturbed medical concept validation (κ = 0.78).

“Claude-Opus-4.6 also shows good alignment with human annotators for unperturbed medical concept validation (𝜅 = 0.78), but its performance drops substantially for perturbed concepts (𝜅 = 0.24) [127].”

• cites [127]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

LLM-as-a-Judge ensembles cluster near the top of agreement rates.

“In addition, we observe that LLM-as-a-Judge ensembles cluster near the top, including a three-judge ensemble (GPT-4, Claude, DeepSeek) that reaches 0.90 on rubric-anchored adequacy and safety items in NSCLC care [22], supporting the view that aggregating heterogeneous judges can reduce idiosyncratic bias.”

The popular application areas in this review are precisely the settings where the bottleneck of expert review is most evident, as each generates open-ended outputs at volumes that exceed the capacity of expert review.

“The popular application areas in this review, including Clinical Decision Support, Clinical NLP, Medical Knowledge & QA, and Medical Communication, are precisely the settings where this bottleneck is most evident, as each generates open-ended outputs at volumes that exceed the capacity of expert review.”

Across 10 examined studies reporting Cohen's κ, values range from 0.59 to 0.88, broadly covering the 'moderate' to 'strong' agreement range under conventional benchmarks.

“Across 10 examined studies reporting Cohen's 𝜅 (Figure 9), values range from 0.59 to 0.88, broadly covering the “moderate” to “strong” agreement range under conventional benchmarks.”

The domain incorporates robustness-and bias-oriented evaluation, such as sensitivity to adversarial inputs, misleading context, and cultural cues.

“The domain also incorporates robustness-and bias-oriented evaluation, such as sensitivity to adversarial inputs, misleading context, and cultural cues [85,116].”

• cites [85]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [116]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

GPT-4o shows lower agreement on mental and behavioral health prompts.

“but shows lower agreement on mental and behavioral health prompts [70].”

• cites [70]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The lowest agreement value is observed in fine-grained biomedical entity linking, where GPT-5.2 reaches 0.66 in distinguishing Correct, Broad, Narrow, and No-relation labels.

“The lowest value is observed in fine-grained biomedical entity linking, where GPT-5.2 reaches 0.66 in distinguishing Correct, Broad, Narrow, and No-relation labels [115]”

• cites [115]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Pairwise or ranking-based metrics (e.g., win rate, rank agreement, and Kendall's τ) are often used to assess whether LLM judges can distinguish higher-quality answers from weaker alternatives.

“Another prominent feature is the use of pairwise or ranking-based metrics (e.g., win rate, rank agreement, and Kendall's 𝜏) are often used to assess whether LLM judges can distinguish higher-quality answers from weaker alternatives [33].”

• cites [33]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Structured factuality checks on Brief Hospital Course narratives reach 0.89 across 13,290 propositions.

“structured factuality checks on Brief Hospital Course narratives reach 0.89 across 13,290 propositions [27]”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Publications continue to be disseminated primarily as preprints (46 arXiv studies among our 134 filtered records).

“The methodology originated in the computer science community, and publications continue to be disseminated primarily as preprints (46 arXiv studies among our 134 filtered records).”

In medical education, 'ExaminerGPT' grades leniently until explicitly instructed to mark more strictly.

“Similar findings appear in medical education, where”

• cites [118]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

This is particularly notable given the moderate-to-high performance of such systems observed across these fields in our review.

“This is particularly notable given the moderate-to-high performance of such systems observed across these fields in our review.”

In biomedical relation extraction, even domain-adapted judges with structured outputs remained limited in recognizing complex relations, ambiguous terminology, and implicit entity connections.

“In biomedical relation extraction, even domain-adapted judges with structured outputs remained limited in recognizing complex relations, ambiguous terminology, and implicit entity connections [71].”

• cites [71]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

LLM judges often struggle to consistently interpret and apply evaluation criteria, conflating surface-level features with substantive quality.

“LLM judges often struggle to consistently interpret and apply evaluation criteria, conflating surface-level features with substantive quality.”

Among open-source model families, LLaMA-3.3-70B (n=5) and LLaMA-3.1-8B (n=4) are the leading Meta models.

“Among open-source model families, LLaMA-3.3-70B(n=5) and LLaMA-3.1-8B(n=4) are the leading Meta models, while Qwen3-32B (n=5) is the most common Alibaba model.”

Across the 134 included studies, LLM judges are most commonly implemented through prompt-based evaluation, augmented by strategies such as ensembles, multi-agent designs, RAG, and fine-tuning.

“Across the 134 included studies, LLM judges are most commonly implemented through prompt-based evaluation, augmented by strategies such as ensembles, multi-agent designs, RAG, and fine-tuning.”

13 studies reported Pearson's r (n=10) or Spearman's rho (n=3) with 95% confidence intervals.

“Forest plot of 13 studies reporting Pearson's 𝑟 (P, 𝑛 = 10) or Spearman's 𝜌 (S, 𝑛 = 3) with 95% confidence intervals, sorted by correlation.”

The sharp increase in 2025 and early 2026 indicates the rapid emergence of LLM-as-a-Judge in the studies.

“The sharp increase in 2025 and early 2026 indicates the rapid emergence of LLM-as-a-Judge in the studies.”

The United Kingdom contributed 14 of 134 included studies (10.4%).

“The geographic distribution of LLM-as-a-Judge research in healthcare reveals a pronounced concentration in the United States, which contributed to 66 of 134 studies (49.3%), followed by China (n=19, 14.2%), the United Kingdom (n=14, 10.4%), Canada (n=10, 7.5%), and Germany (n=8, 6.0%) (Figure 11).”

The median agreement rate between LLM-as-a-Judge and human experts is 0.83 across 13 examined studies.

“The dotted line marks the median agreement rate (0.83), and the dashed line marks the mean agreement rate (0.83) across the 13 examined studies.”

The mean Cohen's kappa between LLM-as-a-Judge and human experts is 0.76 across 10 examined studies.

“The dotted line marks the median Cohen's 𝜅 (0.78), and the dashed line marks the mean Cohen's 𝜅 (0.76) across the 10 examined studies.”

The United States contributed 66 of 134 included studies (49.3% in the text; 48.9% in Figure 11).

“The geographic distribution of LLM-as-a-Judge research in healthcare reveals a pronounced concentration in the United States, which contributed to 66 of 134 studies (49.3%), followed by China (n=19, 14.2%), the United Kingdom (n=14, 10.4%), Canada (n=10, 7.5%), and Germany (n=8, 6.0%) (Figure 11).”

• Also stated in: § Additional information · plain

  “The United States contributed the largest share (n=66, 48.9%), followed by China (n=19), the United Kingdom (n=14), Canada (n=10), and Germany (n=8).”

To our knowledge, no large-scale controlled comparison has systematically varied judge architecture (e.g., single judge vs. ensemble vs. multi-agent), prompting strategy (e.g., rubric-only vs. CoT vs. few-shot), or grounding source (e.g., closed-book vs. retrieval-augmented) while holding the clinical task fixed.

“To our knowledge, no large-scale controlled comparison has systematically varied judge architecture (e.g., single judge vs. ensemble vs. multi-agent), prompting strategy (e.g., rubric-only vs. CoT vs. few-shot), or grounding source (e.g., closed-book vs. retrieval-augmented) while holding the clinical task fixed.”

The mean agreement rate between LLM-as-a-Judge and human experts is 0.83 across 13 examined studies.

“The dotted line marks the median agreement rate (0.83), and the dashed line marks the mean agreement rate (0.83) across the 13 examined studies.”

The mean correlation between LLM-as-a-Judge and human expert ratings is 0.68 across 13 examined studies.

“The dotted line marks the median correlation value (0.69), and the dashed line marks the mean correlation value (0.68) across the 13 examined studies.”

Overall, open-source models appear in 59 studies (45.4%).

“Overall, closed-source models appear in 104 studies (80.0%) and open-source models in 59 studies (45.4%).”

The median Cohen's kappa between LLM-as-a-Judge and human experts is 0.78 across 10 examined studies.

“The dotted line marks the median Cohen's 𝜅 (0.78), and the dashed line marks the mean Cohen's 𝜅 (0.76) across the 10 examined studies.”

Among 277 unique institutions represented across the included studies (241 institutions with at least one collaboration per Figure 12), Harvard University was the most prolific contributor (n=8 studies).

“Among 277 unique institutions represented across the included studies, Harvard University was the most prolific contributor (n=8 studies), followed by Stanford University, the University of Illinois Urbana-Champaign, and Ant Group (n=5 each), and MIT, the University of Colorado Anschutz, the National University of Singapore, and Mohamed bin Zayed University of Artificial Intelligence (n=4 each).”

• Also stated in: § Additional information · plain

  “Harvard University (n=8), Stanford University (n=5), UIUC (n=5), and Ant Group (n=5) were the most prolific contributors.Based on 241 institutions with at least one collaboration across 134 studies.”

China contributed 19 of 134 included studies (14.2%).

“The geographic distribution of LLM-as-a-Judge research in healthcare reveals a pronounced concentration in the United States, which contributed to 66 of 134 studies (49.3%), followed by China (n=19, 14.2%), the United Kingdom (n=14, 10.4%), Canada (n=10, 7.5%), and Germany (n=8, 6.0%) (Figure 11).”

Overall, closed-source models appear in 104 studies (80.0%).

“Overall, closed-source models appear in 104 studies (80.0%) and open-source models in 59 studies (45.4%).”

33 (25.4%) of 130 studies with identifiable model types use both closed-source and open-source models.

“Among 130 studies with identifiable model types (4 studies do not clarify the judge models), 71 (54.6%) use closed-source models exclusively, 26 (20.0%) use open-source models exclusively, and 33 (25.4%) use both (Figure 13).”

International collaboration was observed in 33 of 134 studies.

“International collaboration was observed in 33 of 134 studies, with the most frequent cross-border partnerships occurring between the USA and Canada (n=4), China and the UK (n=4), and the USA and the UK (n=3).”

26 (20.0%) of 130 studies with identifiable model types use open-source models exclusively.

“Among 130 studies with identifiable model types (4 studies do not clarify the judge models), 71 (54.6%) use closed-source models exclusively, 26 (20.0%) use open-source models exclusively, and 33 (25.4%) use both (Figure 13).”

OpenAI models were used in 90 studies.

“OpenAI (n=90 studies), Google (n=35), Meta (n=25), Anthropic (n=27), Alibaba (n=24), and DeepSeek (n=13).”

Bimonthly trend analysis shows that closed-source models dominate throughout the study period.

“Bimonthly trend analysis shows that closed-source models dominate throughout the study period, but open-source adoption accelerates markedly from mid-2025 onward.”

Most evidence for LLM-as-a-Judge is currently retrospective, relying on archived cases, benchmark datasets, or post-hoc comparison with expert annotations.

“Most evidence for LLM-as-a-Judge is currently retrospective, relying on archived cases, benchmark datasets, or post-hoc comparison with expert annotations.”

Open-source adoption accelerates markedly from mid-2025 onward.

“Bimonthly trend analysis shows that closed-source models dominate throughout the study period, but open-source adoption accelerates markedly from mid-2025 onward.”

Among 130 studies with identifiable model types, 71 (54.6%) use closed-source models exclusively.

“Among 130 studies with identifiable model types (4 studies do not clarify the judge models), 71 (54.6%) use closed-source models exclusively, 26 (20.0%) use open-source models exclusively, and 33 (25.4%) use both (Figure 13).”

Qwen3-32B (n=5) is the most common Alibaba model.

“Among open-source model families, LLaMA-3.3-70B(n=5) and LLaMA-3.1-8B(n=4) are the leading Meta models, while Qwen3-32B (n=5) is the most common Alibaba model.”

Canada contributed 10 of 134 included studies (7.5%).

“The geographic distribution of LLM-as-a-Judge research in healthcare reveals a pronounced concentration in the United States, which contributed to 66 of 134 studies (49.3%), followed by China (n=19, 14.2%), the United Kingdom (n=14, 10.4%), Canada (n=10, 7.5%), and Germany (n=8, 6.0%) (Figure 11).”

For Anthropic, Claude-3.7-Sonnet (n=5), Claude-3-Haiku (n=4), and Claude-3.5-Sonnet (n=4) are the most frequently used models.

“For Anthropic, Claude-3.7-Sonnet(n=5), Claude-3-Haiku (n=4), and Claude-3.5-Sonnet(n=4) are the most frequently used models.”

Germany contributed 8 of 134 included studies (6.0%).

“The geographic distribution of LLM-as-a-Judge research in healthcare reveals a pronounced concentration in the United States, which contributed to 66 of 134 studies (49.3%), followed by China (n=19, 14.2%), the United Kingdom (n=14, 10.4%), Canada (n=10, 7.5%), and Germany (n=8, 6.0%) (Figure 11).”

The median correlation between LLM-as-a-Judge and human expert ratings is 0.69 across 13 examined studies.

“The dotted line marks the median correlation value (0.69), and the dashed line marks the mean correlation value (0.68) across the 13 examined studies.”

The five most prolific countries accounted for over 86% of all publications.

“These five countries accounted for over 86% of all publications.”

Major research domains include Clinical Decision Support, Clinical NLP, Medical Knowledge & QA, and Medical Communication.

“Major research domains include Clinical Decision Support, Clinical NLP, Medical Knowledge & QA, and Medical Communication.”

Research contributions spanned 30 countries across six continents.

“Research contributions spanned 30 countries across six continents, with notable representation from Asia-Pacific nations including Japan (n=6), the UAE (n=6), South Korea (n=5), and Singapore (n=4), reflecting the global interest in applying LLMs for clinical evaluation.”

Prior work suggests LLM-based judges can approximate human judgments in reasoning, summarization, and dialogue tasks.

“Prior work suggests that LLM-based judges can approximate human judgments in reasoning, summarization, and dialogue tasks [66,105].”

• cites [66]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [105]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Errors from LLM judges may have substantial downstream consequences if they shape deployment decisions.

“Such errors may have substantial downstream consequences if they shape deployment decisions [65].”

• cites [65]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Many healthcare quality indicators are embedded in narrative clinical notes rather than structured fields.

“This limitation is particularly important in healthcare, where many quality indicators are embedded in narrative clinical notes rather than structured fields [94].”

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Automated metrics such as BLEU, ROUGE, and BERTscore measure lexical similarity rather than medical correctness or reasoning validity.

“Automated metrics (e.g., BLEU [106], ROUGE [83], BERTscore [163]) measure lexical similarity rather than medical correctness or reasoning validity.”

• cites [106]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [83]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [163]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The field has not yet fully addressed macro-level healthcare challenges, including regulatory evaluation and population health assessment.

“Their limited representation also suggests that the field has not yet fully addressed macro-level healthcare challenges, including regulatory evaluation and population health assessment.”

LLM-as-a-Judge is primarily used in decision-critical and text-intensive settings, where scalable and consistent evaluation is needed.

“This distribution shows that LLM-as-a-Judge is primarily used in decision-critical and text-intensive settings, where scalable and consistent evaluation is needed.”

This trend indicates that automated evaluation has quickly become an important methodological direction for assessing AI-generated clinical content.

“This trend indicates that automated evaluation has quickly become an important methodological direction for assessing AI-generated clinical content.”

Current research gives greater attention to front-end decision-making, such as diagnosis and triage, than to downstream intervention validation and safety monitoring.

“This imbalance suggests that current research gives greater attention to front-end decision-making, such as diagnosis and triage, than to downstream intervention validation and safety monitoring, despite the importance of these areas for real-world clinical deployment.”

These studies indicate the potential value of LLM-based evaluation for system-level decision-making and policy evaluation.

“Although limited in number, these studies indicate the potential value of LLM-based evaluation for systemlevel decision-making and policy evaluation.”

LLM-as-a-Judge is most often applied to holistic, language-intensive evaluation tasks, where traditional rule-based or metric-based methods may be insufficient.

“This pattern suggests that LLMas-a-Judge is most often applied to holistic, language-intensive evaluation tasks, where traditional rule-based or metric-based methods may be insufficient.”

Agreement is task-dependent within the same judge family.

“Agreement is also task-dependent within the same judge family: GPT-4o reaches 0.88 in medical image quality assessment [88], but shows lower agreement on mental and behavioral health prompts [70].”

Raw agreement can be sensitive to the reference standard.

“Finally, raw agreement can be sensitive to the reference standard: in one study, the same judge that reached 0.83 agreement with at least one expert dropped to 0.51 against majority-of-experts judgments [23], highlighting how expert disagreement and subjective criteria can affect apparent judge reliability.”

Well-scoped tasks with anchored rubrics generally outperform open-ended generation.

“Well-scoped tasks with anchored rubrics generally outperform open-ended generation: structured factuality checks on Brief Hospital Course narratives reach 0.89 across 13,290 propositions [27], whereas mental-health counseling rubrics with more subjective language register 0.73 [81].”

Clinical errors are not always captured by accuracy alone.

“This broader evaluation is important because clinical errors are not always captured by accuracy alone.”

LLM-as-a-Judge has emerged as a practical response to this gap, offering a way to assess content along dimensions that are difficult to formalize.

“LLM-as-a-Judge has emerged as a practical response to this gap, offering a way to assess content along dimensions (e.g., reasoning quality, factual consistency, completeness, safety, empathy, and clinical usefulness) that are difficult to formalize [21,29,36,45,81,88,153].”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [81]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [88]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [153]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The evaluation gap of fixed-answer benchmarks and conventional metrics for unstructured clinical data has become more pronounced as clinical AI systems increasingly produce open-ended outputs.

“As these systems increasingly produce open-ended outputsdiagnostic narratives, counseling responses, discharge summaries, and patient-facing explanationsthe evaluation gap of fixed-answer benchmarks and conventional metrics for unstructured clinical data have become more pronounced.”

Prevalence-corrected reliability can be substantially lower when one rating category dominates.

“showing that prevalence-corrected reliability can be substantially lower when one rating category dominates.”

LLM-as-a-Judge offers a path through this trade-off by approximating rubric-based expert judgment on free-text outputs at substantially reduced time and cost.

“LLM-as-a-Judge offers a path through this trade-off by approximating rubric-based expert judgment on free-text outputs at substantially reduced time and cost.”

In Medical Knowledge & QA, LLM-as-a-Judge evaluation places greater emphasis on reasoning quality, factual correctness, and answer justification, reflecting the explanatory nature of medical QA tasks.

“In this domain, LLM-as-a-Judge evaluation places greater emphasis on reasoning quality, factual correctness, and answer justification, reflecting the explanatory nature of medical QA tasks.”

This granular approach enables targeted assessment of hallucination and misinformation beyond a single overall score.

“This granular approach enables targeted assessment of hallucination and misinformation beyond a single overall score.”

More subjective or multi-dimensional rubric tasks tend to show lower reliability.

“More subjective or multi-dimensional rubric tasks tend to show lower reliability; for example, four-class clinical risk grading in MedVAL remains in the moderate range (𝜅 = 0.67) [1].”

The rapid growth of LLM-as-a-Judge research in healthcare suggests a broad change in how clinical AI systems are evaluated.

“The rapid growth of LLM-as-a-Judge research in healthcare (Figure 3) suggests a broad change in how clinical AI systems are evaluated.”

Aggregating heterogeneous judges can reduce idiosyncratic bias.

“supporting the view that aggregating heterogeneous judges can reduce idiosyncratic bias.”

Compared with Clinical Decision Support, metrics in Clinical NLP are more text-centric and decomposition-driven, with emphasis on fine-grained rubric design and assessment of document-level clinical workflows.

“Overall, compared with Clinical Decision Support, metrics in Clinical NLP are more text-centric and decomposition-driven, with emphasis on fine-grained rubric design and assessment of document-level clinical workflows.”

Expert disagreement and subjective criteria can affect apparent judge reliability.

“highlighting how expert disagreement and subjective criteria can affect apparent judge reliability.”

Across examined studies reporting quantitative validation, the evidence is encouraging.

“Across examined studies reporting quantitative validation (Section 3.4), the evidence is encouraging.”

In safety-critical settings, false-negative judgments are especially important because they may allow unsafe outputs to pass without human review.

“In safety-critical settings, false-negative judgments are especially important because they may allow unsafe outputs to pass without human review [23,140].”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [140]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Without regular updates or re-validation, an LLM judge may continue to apply outdated criteria or produce evaluations that no longer align with current standards of care.

“Without regular updates or re-validation, an LLM judge may continue to apply outdated criteria or produce evaluations that no longer align with current standards of care [36,155].”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [155]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Clinical standards change over time as new guidelines, therapies, evidence, and local workflows emerge.

Bias arising from judges evaluating outputs generated by models in the same family has been examined in prior work.

“Bias arising from judges evaluating outputs generated by models in the same family has been examined in prior work [73].”

• cites [73]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

For safety-sensitive tasks such as psychosocial risk assessment or treatment recommendation, multi-agent or persona-based designs can expose disagreement that would be hidden in a single score.

“For safety-sensitive tasks such as psychosocial risk assessment or treatment recommendation, multi-agent or persona-based designs can expose disagreement that would be hidden in a single score [23,90].”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [90]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

LLM-as-a-Judge is informative when the rubric is tied to the specific clinical output.

“This suggests that LLM-as-a-Judge is informative when the rubric is tied to the specific clinical output, such as PDSQI-9 for documentation quality [29], OSCE-style criteria for clinical skills [159], or atomic-claim factuality assessment against EHR evidence [27].”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [159]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

These results indicate that strong alignment with experts is achievable, particularly when the task is well scoped and the rubric is explicit.

“These results indicate that strong alignment with experts is achievable, particularly when the task is well scoped and the rubric is explicit.”

The risks are amplified when judging workflows involve protected health information.

“The risks are amplified when judging workflows involve protected health information.”

Errors in judging diagnostic reasoning or treatment recommendations may propagate unsafe clinical decisions.

“Errors in judging diagnostic reasoning or treatment recommendations may propagate unsafe clinical decisions [36,140,146]”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [140]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [146]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

This limits the portability and external validity of LLM-as-a-Judge frameworks.

“This limits the portability and external validity of LLM-as-a-Judge frameworks.”

A GPT-based judge evaluating GPT-generated clinical text may assign high scores to outputs containing errors that GPT-family models systematically fail to recognize, not because such errors are absent, but because both systems lack the capacity to identify them reliably.

“As a result, a GPT-based judge evaluating GPT-generated clinical text may assign high scores to outputs containing errors that GPT-family models systematically fail to recognize, not because such errors are absent, but because both systems lack the capacity to identify them reliably.”

Long-form medical Q&A benchmarks suggest that correctness alone is insufficient for assessing clinical answer quality.

“Similarly, long-form medical Q&A benchmarks suggest that correctness alone is insufficient for assessing clinical answer quality [53].”

• cites [53]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

These biases could lead to uneven evaluation quality across patient subgroups, institutions, or care environments.

“These biases could lead to uneven evaluation quality across patient subgroups, institutions, or care environments [52,146].”

• cites [52]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [146]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Generic prompts that ask for overall quality are less likely to produce clinically interpretable judgments.

“By contrast, generic prompts that ask for overall”

A judge that performs well on a benchmark or within one hospital system may not perform equally well in another setting.

“As a result, a judge that performs well on a benchmark or within one hospital system may not perform equally well in another setting [1,31,146].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [31]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [146]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Healthcare settings vary widely in patient populations, disease prevalence, EHR systems, documentation practices, clinical workflows, local policies, resources, and language use.

A related failure mode concerns shallow domain reasoning beneath fluent biomedical language.

“A related failure mode concerns shallow domain reasoning beneath fluent biomedical language.”

When both the generator and the evaluator belong to the same model family, they often share similar training data distributions, inductive biases, and knowledge gaps.

“Specifically, when both the generator and the evaluator belong to the same model family, they often share similar training data distributions, inductive biases, and knowledge gaps.”

LLM judges may inherit demographic, linguistic, and socioeconomic biases from their underlying foundation models.

“A further concern is that LLM judges may inherit demographic, linguistic, and socioeconomic biases from their underlying foundation models.”

Current LLM judges remain sensitive to prompt design and language resources.

“These findings indicate that current LLM judges remain sensitive to prompt design and language resources, raising concerns for both reproducibility and equity across clinical settings.”

This creates a form of correlated evaluation bias in which shared blind spots can artificially inflate perceived performance.

“This creates a form of correlated evaluation bias in which shared blind spots can artificially inflate perceived performance.”

Clinical risks remain a major barrier to deploying LLM judges in healthcare.

“Clinical risks remain a major barrier to deploying LLM judges in healthcare.”

In clinical document summarization, judges are similarly unreliable in detecting factual hallucinations in generated outputs.

“In clinical document summarization, judges are similarly unreliable in detecting factual hallucinations in generated outputs [140].”

• cites [140]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

For knowledge-intensive judgments, retrieval can supply guidelines, drug information, or patient records directly to the judge.

“For knowledge-intensive judgments, retrieval can supply guidelines, drug information, or patient records directly to the judge [121,155].”

• cites [121]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [155]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Automated evaluation, in this case, no longer provides an independent check on generation quality.

“Automated evaluation, in this case, no longer provides an independent check on generation quality.”

Several studies show that evaluator models can hallucinate as well.

“Prior work on hallucination has focused primarily on generation models inventing facts, but several studies show that evaluator models can hallucinate as well.”

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [140]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Inaccuracies in evaluating clinical notes or discharge summaries may compromise communication, billing, and medico-legal records.

“while inaccuracies in evaluating clinical notes or discharge summaries may compromise communication, billing, and medico-legal records [23,126].”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [126]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Overly favorable or overconfident judgments may create false reassurance and obscure unsafe model outputs, particularly when healthcare LLMs generate fluent but factually incorrect or clinically inappropriate responses.

“Importantly, overly favorable or overconfident judgments may create false reassurance and obscure unsafe model outputs, particularly when healthcare LLMs generate fluent but factually incorrect or clinically inappropriate responses [36,126,140].”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [126]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [140]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

When both the generator and the evaluator are LLMs, their errors can reinforce rather than correct one another.

“This poses a particular safety concern: when both the generator and the evaluator are LLMs, their errors can reinforce rather than correct one another.”

Patient-facing use raises ethical and legal concerns.

“Patient-facing use raises ethical and legal concerns.”

The observed variation in judge performance may reflect differences in study design and reporting quality, rather than differences in the capability of LLM judges.

“As a result, the observed variation in judge performance may reflect differences in study design and reporting quality, rather than differences in the capability of LLM judges.”

Many studies illustrate the alignment with human experts but provide limited information on operational dimensions of LLM-as-a-Judge deployment, such as inference cost, latency, reproducibility, privacy safeguards, and integration with existing clinical workflows.

“Many studies illustrate the alignment with human experts but provide limited information on operational dimensions of LLM-as-a-Judge deployment, such as inference cost, latency, reproducibility, privacy safeguards, and integration with existing clinical workflows.”