Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study

Classified as probably not a substantial claim. May combine two claims.

LLMs are capable of processing large volumes of text and generating human-like summaries and interpretations.

“LLMs are capable of processing large volumes of text and generating human-like summaries and interpretations [2].”

• cites [2] Blankemeier, 2024, Adapted large language models can outperform medical experts in clinical text summarization, Nat. Med. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Farquhar S. et al. propose a method based on semantic entropy, which provides a partial solution for identifying errors associated with confabulations.

“For example, Farquhar S. et al. in the article [37] propose a method based on semantic entropy, which provides a partial solution for identifying errors associated with confabulations.”

• cites [37] Farquhar, 2024, Detecting hallucinations in large language models using semantic entropy, Nature sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This method demonstrates high agreement with expert evaluations in assessing answer quality, achieving a Pearson correlation coefficient of 0.917, compared to 0.817 for GPT-4.

“This method demonstrates high agreement with expert evaluations in assessing answer quality, achieving a Pearson correlation coefficient of 0.917, compared to 0.817 for GPT-4 [38].”

• cites [38] Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv. sourcePartly supported

  Table 2 reports the stated Pearson correlations: 0.917 for PoLL and 0.817 for GPT-4. But Section 4.2 says these measure agreement between model rankings and the crowd-sourced Chatbot Arena leaderboard, not agreement with expert evaluations of individual answers. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Methods for automated detection of hallucinations based on pre-trained models have been proposed.

“Methods for automated detection of hallucinations based on pre-trained models have been proposed [35,36,37].”

• cites [35] Urlana, A., Kanumolu, G., Kumar, C.V., Garlapati, B.M., and Mishra, R. (2025). HalluCounter: Reference-free LLM Hallucination Detection in the Wild!. arXiv. sourceSupported
• cites [36] Park, S., Du, X., Yeh, M.-H., Wang, H., and Li, Y. (2025). Steer LLM Latents for Hallucination Detection. arXiv. sourceSupported
• cites [37] Farquhar, 2024, Detecting hallucinations in large language models using semantic entropy, Nature sourceSupported

May combine two claims.

Modern LLMs suffer from "contextual forgetting": as the text length increases, model performance worsens and relevant information gets lost.

“As is well known, modern LLMs suffer from “contextual forgetting”: as the text length increases, model performance worsens and relevant information gets lost [32,33,34].”

• cites [32] Shi, J., Ma, Q., Liu, H., Zhao, H., Hwang, J.-N., and Li, L. (2025). Explaining context length scaling and bounds for language models. arXiv. sourcePartly supported

  Section G.2 says LLMs can perform worse when a long context contains too much irrelevant information, and Section 3 reports increased validation loss beyond an optimal context length. But the Introduction also says longer context can initially improve performance; the paper does not establish that performance generally worsens as text grows or that relevant information is lost. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [33] Li, J., Wang, M., Zheng, Z., and Zhang, M. (2023). Loogle: Can long-context language models understand long contexts?. arXiv. sourceSupported
• cites [34] Gao, Y., Xiong, Y., Wu, W., Huang, Z., Li, B., and Wang, H. (2025). U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack. arXiv. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Verga P. and colleagues propose an approach that employs a diverse panel of models.

“Verga P. and colleagues propose an approach that employs a diverse panel of models.”

• cites [38] Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv. sourceSupported

May combine two claims.

The Gemma model (27 billion) demonstrated lower overall performance on average compared to the Llama model (8 billion), with the exception of the 'Grammar and Terminology' criterion.

“Interestingly, the Gemma model (27 billion) demonstrated lower overall performance on average compared to the Llama model (8 billion), with the exception of the “Grammar and Terminology” criterion.”

May combine two claims.

Neither automatic evaluation metrics nor LLM judges are reliable in detecting factual errors and semantic distortions (hallucinations).

“Nevertheless, neither the evaluation metrics nor LLM judges are reliable in detecting factual errors and semantic distortions (hallucinations).”

• Also stated in: § 3. Results · plain

  “However, both LLMs identified only two of the nine hallucinations detected by the experts.”

• AssumptionRests on an assumption: Any LLM poses risks of hallucinations, and input data and evaluation outputs may be biased due to algorithmic limitations and lack of diversity in training datasets.

May combine two claims.

Automatic evaluation metrics and LLM judges' scores showed low correlation with each other, indicating a weak relationship.

“The results demonstrated low correlation between the metrics, which indicates a weak relationship.”

May combine two claims.

Compared to Qwen (32 billion), the Gemma model (12 billion) showed significantly lower relevance and completeness while outperforming in grammar and terminology, with similarly low hallucinations rate.

“Compared to Qwen (32 billion), the Gemma model (12 billion) showed significantly lower relevance and completeness while outperforming in grammar and terminology, with similarly low hallucinations rate.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The average summary generation time was 25 s; automated evaluation by Qwen3-14b took on average 200 s, and by Mistral-small-24b took 85 s.

“The average summary generation time was 25 s. In contrast, the automated evaluation by Qwen3-14b took, on average, 200 s, and that by Mistral-small-24b took 85 s.”

The quote only partly states this claim.

LLM judges effectively identify significant shortcomings in the summaries.

“The LLM judges effectively identify significant shortcomings in the summaries; however, they frequently conflate or misapply the evaluation criteria.”

May combine two claims.

While the original EHR contained a referral for an abdominal ultrasound, the corresponding summary stated "Abdominal ultrasound: not performed".

“While the original EHR contained a referral for an abdominal ultrasound, the corresponding summary stated “Abdominal ultrasound: not performed”—a claim potentially misrepresenting the actual course of events.”

Probably not a substantial claim: may not pass S4 (based).

Hallucinations occur in LLMs even at zero temperature for summary generation.

“It should also be noted that neither the proposed automatic evaluation metrics nor LLM judges are capable of reliably detecting hallucinations, which occur in LLMs even at zero temperature for summary generation.”

Classified as probably not a substantial claim.

The final score was determined by expert consensus.

“The final score was determined by expert consensus.”

May combine two claims.

Inherent LLM biases impact the performance of LLM-as-a-judge, leading to biased assessments, preference for redundancy at the expense of accuracy, overestimation of decision quality, and similar issues.

“Therefore, biased assessments, preference for redundancy at the expense of accuracy, overestimation of the decision quality, and similar issues [24] prevent LLM judges from being considered reliable, autonomous evaluation tools, especially in the highly sensitive medical field.”

• cites [24] Van Schaik, T.A., and Pugh, B. (2024, January 14–18). A field guide to automatic evaluation of LLM-generated summaries. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24), Washington, DC, USA. sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Any LLM poses risks of hallucinations, and input data and evaluation outputs may be biased due to algorithmic limitations and lack of diversity in training datasets.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Russian Federation, LLM-based methods (like all AI technologies) are classified in medicine as Class III high-risk technologies.

“Therefore, in Russian Federation LLM-based methods (like all AI technologies) are classified in medicine as Class III high-risk technologies [14].”

• cites [14] (2025, November 21). Order of the Ministry of Health of the Russian Federation No. 4n of June 6, 2012 “On the Approval of the Nomenclature Classification of Medical Devices”. Available online: https://normativ.kontur.ru/document?moduleId=1&documentId=501045.Not checked yet

  Not checked in the free run.

May combine two claims.

Qwen3-14b LLM metrics were excluded from further analysis due to lower correlation with expert assessment compared to Mistral-small-24b, in addition to bias manifested as score overestimation for models of its class.

“Therefore, the Qwen3-14b LLM metrics were excluded from further analysis due to lower correlation with the expert assessment compared to Mistral-small-24b (Figure 1), in addition to the bias manifested as score overestimation for the models of its class (Table 3).”

Probably not a substantial claim: may not pass S2 (checkable).

The combination of LLMs and automatic evaluation metrics can underpin medical decision support systems by performing initial evaluations and highlighting potential shortcomings for expert review.

“Nevertheless, even now, the combination of LLMs and the automatic evaluation metrics can underpin medical decision support systems by performing initial evaluations and highlighting potential shortcomings for expert review.”

Classified as probably not a substantial claim. May combine two claims.

No specialized fine-tuning of the models was performed, which reflects a practical scenario where end-users deploy off-the-shelf models without adaptation, improving reproducibility and generalizability.

“No specialized fine-tuning of the models was performed. It reflects a practical scenario where end-users deploy off-the-shelf models without adaptation, which improves the reproducibility and generalizability of our methodology.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

All models were run at zero temperature, which minimizes generation stochasticity and ensures maximum correspondence between generated summaries and EHR data.

“All models were run at zero temperature, which minimizes generation stochasticity and ensures maximum correspondence between the generated summaries and the EHR data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One advantage of the LLM-as-a-judge approach is that the method does not require reference summaries.

“One advantage of the LLM-as-a-judge approach is that the method does not require reference summaries.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A. Ahmed et al. propose enhancing low-parameter models for medical contexts through advanced prompting techniques, improving their accuracy and relevance.

“Similarly, A. Ahmed et al. propose enhancing low-parameter models for medical contexts through advanced prompting techniques, improving their accuracy and relevance [28].”

• cites [28] Ahmed, A., Hou, M., Xi, R., Zeng, X., and Shah, S.A. (2024, January 13–17). Prompt-eng: Healthcare prompt engineering: Revolutionizing healthcare applications with precision prompts. Proceedings of the Companion Proceedings of the ACM Web Conference 2024, Singapore. sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Independent fine-tuning of models requires substantial resources, primarily due to expert data annotation.

May combine two claims.

BERTScore's stronger correlation with expert scores is due to its accounting for synonymous substitutions and syntactic paraphrases, demonstrating robustness to lexical diversity.

“Moreover, the BERTScore metric demonstrated a stronger correlation with the expert scores, as it takes into account possible synonymous substitutions and syntactic paraphrases, demonstrating robustness to lexical diversity.”

May combine two claims.

Summarization quality assessment requires a multi-criteria approach, and model performance can vary depending on the task and the priority of a particular criterion.

“Thus, summarization quality assessment requires a multi-criteria approach, and the model performance can vary depending on the task and, accordingly, the priority of a particular criterion.”

Probably not a substantial claim: may not pass S4 (based).

The validity criterion for LLM-as-a-judge is adequate summary ranking, i.e., an LLM judge should assign lower scores to lower-quality summaries.

“The validity criterion for LLM-as-a-judge is the adequate summary ranking. In other words, an LLM judge should assign lower scores to lower-quality summaries.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study protocol was approved by the IEC of the Moscow Regional Branch of the Russian Society of Roentgenographers and Radiologists (MRB RSRR) (Protocol No. 6 dated 19 June 2025).

“The study protocol was approved by the IEC of the Moscow Regional Branch of the Russian Society of Roentgenographers and Radiologists (MRB RSRR) (Protocol No. 6 dated 19 June 2025).”

Classified as probably not a substantial claim. May combine two claims.

The reference summaries were created by clinicians and validated by a radiologist.

“The reference summaries were created by clinicians and validated by a radiologist.”

Classified as probably not a substantial claim. May combine two claims.

A standalone assessment requires using both metric groups, which complement each other rather than duplicate, to provide a comprehensive summary quality score.

“Therefore, a standalone assessment requires using both metric groups, which do not duplicate but rather complement each other, providing a comprehensive summary quality score.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The low correlation between expert scores and LLM-as-a-judge may be explained by both minor differences in the quality of summaries for these criteria and limitations of expert review.

“The low correlation between the expert scores and LLM-as-a-judge may be explained by both minor differences in the quality of summaries for these criteria and limitations of expert review.”

• AssumptionRests on an assumption: Expert evaluations are inherently subjective, being grounded in individual professional experience.

May combine two claims.

This study employed only low-parameter, local, open-access general-purpose models without fine-tuning.

“This study employed only low-parameter, local, open-access general-purpose models without fine-tuning, as they present the most straightforward and transparent option for practical deployment.”

Probably not a substantial claim: may not pass S4 (based).

Any changes in expert evaluation methodology-such as criteria, rating scales, or volume and structure of the source text-will influence the resulting metrics.

“It should be noted that any changes in expert evaluation methodology—such as criteria, rating scales, or volume and structure of the source text—will influence the resulting metrics.”

Probably not a substantial claim: may not pass S2 (checkable).

Even at the current stage, the combination of LLMs and the automatic evaluation metrics can provide the basis for medical decision support systems to improve experts' performance by drawing attention to identified deficiencies in medical summaries.

“Nevertheless, even at the current stage, the combination of LLMs and the automatic evaluation metrics can provide the basis for medical decision support systems to improve experts’ performance by drawing attention to identified deficiencies in medical summaries”

May combine two claims.

When the reference text is produced by an expert, BERTScore measures the degree of conformity to expert expectations through domain-specific embeddings, whereas Mistral Small 24B cannot compete with the target metrics in settings requiring high factual accuracy.

“When the reference text is produced by an expert, BERTScore measures the degree of conformity to expert expectations through domain-specific embeddings, whereas Mistral Small 24B, being an open quantized model with limited domain adaptation, cannot compete with the target metrics in settings requiring high factual accuracy.”

May combine two claims.

Full automation of the assessment of medical text summarization performance remains an open question, primarily because neither automatic evaluation metrics nor LLM judges are capable of reliably identifying factual errors and semantic distortions (hallucinations).

“Despite significant progress in the field of LLMs, full automation of the assessment of the medical text summarization performance remains an open question, primarily because neither automatic evaluation metrics nor LLM judges are capable of reliably identifying factual errors and semantic distortions (hallucinations).”

Classified as probably not a substantial claim. May combine two claims.

The research was conducted on a limited dataset using a constrained set of LLMs for both summarization and evaluation.

“The research was conducted on a limited dataset using a constrained set of LLMs for both summarization and evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The semantic entropy method is not tied to a specific task and could potentially complement the proposed integrated score for the quality of medical text summaries.

“The method is not tied to a specific task and could potentially complement the proposed integrated score for the quality of medical text summaries.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Implementation of tailored hallucination detection methods and the use of LLMs with a larger number of parameters or more highly specialized LLMs trained primarily on medical data will improve the reliability, validity, and accuracy.

“Implementation of tailored hallucination detection methods and the use of LLMs with a larger number of parameters or more highly specialized LLMs trained primarily on medical data will improve the reliability, validity, and accuracy.”

Probably not a substantial claim: may not pass S2 (checkable).

The key reason for BERTScore's higher correlation with expert opinions compared to LLM-as-a-judge may lie in the fundamental difference in scoring approaches.

“The key reason may lie in the fundamental difference in the approaches to the scoring.”

• AssumptionRests on an assumption: An LLM judge remains a “generalist” whose decisions are influenced by the vast array of data on which it was trained.
• AssumptionRests on an assumption: Models simulate reasoning but suffer from high entropy and are biased toward superficial text attributes.
• AssumptionRests on an assumption: The quantized version of Mistral Small 24B reduces calculation accuracy and decreases sensitivity to subtle semantic differences.

Probably not a substantial claim: may not pass S2 (checkable).

The development of an autonomous quality assessment tool does not require full correlation with expert opinion.

“It is worth noting that the development of an autonomous quality assessment tool does not require full correlation with expert opinion, which appears methodologically challenging due to subjectivity of the latter.”

• AssumptionRests on an assumption: Expert evaluations are inherently subjective, being grounded in individual professional experience.

Probably not a substantial claim: may not pass S4 (based).

BERTScore is a "special tool" optimized for one specific problem: determining the extent to which the semantics of a single text corresponds to a larger semantic space optimized for the medical domain.

“Thus, BERTScore is a “special tool” optimized for one specific problem: determining the extent to which the semantics of a single text corresponds to a larger semantic space optimized for the medical domain.”

• AssumptionRests on an assumption: The BERTScore architecture is designed to process text fragments using semantic frame embeddings, maintaining accuracy even with large context lengths.