Evaluation of a retrieval-augmented generation system using a Japanese Institutional Nuclear Medicine Manual and large language model-automated scoring
Open accessAbstract
Recent advances in large language models (LLMs) enable domain-specific question answering using external knowledge. However, addressing information that is not included in training data remains a challenge, particularly in nuclear medicine, where examination protocols are frequently updated and vary across institutions. In this study, we developed a retrieval-augmented generation (RAG) system using 40 internal manuals from a single Japanese hospital, each corresponding to a different examination in nuclear medicine. These institution-specific documents were segmented and indexed using a hybrid retrieval strategy combining dense vector search (text-embedding-3-small) and sparse keyword search (BM25). GPT-3.5 and GPT-4o were used with the OpenAI application programming interface (API) for response generation. The quality of the generated answers was assessed using a four-point Likert scale by three certified radiological technologists, of which one held an additional certification in nuclear medicine and another held an additional certification in medical physics. Automated evaluation was conducted using RAGAS metrics, including factual correctness and context recall. The GPT-4o model combined with hybrid retrieval achieved the highest performance, as per expert evaluations. Although traditional string-based metrics such as ROUGE and the Levenshtein distance poorly align with human ratings, RAGAS provided consistent rankings across system configurations, despite showing only a modest correlation with manual scores. These findings demonstrate that integrating examination-specific institutional manuals into RAG frameworks can effectively support domain-specific question answering in nuclear medicine. Moreover, LLM-based evaluation methods such as RAGAS may serve as practical tools to complement expert reviews in developing healthcare-oriented artificial intelligence systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In a nuclear medicine Q&A system, string-based metrics (ROUGE / Levenshtein distance) did not correlate with radiologists' assessments of answer quality.
“For instance, in a nuclear medicine Q&A system, string-based metrics (ROUGE / Levenshtein distance) did not correlate with radiologists' assessments of answer quality [6].”
From Evaluating Metrics for Safety with LLM-as-Judges (2025) · cites this paper as [6]
In Results, Table 4 reports positive correlations between human answer-quality ratings and both ROUGE (r = 0.262–0.525) and Levenshtein distance (r = 0.332–0.489), rather than no correlation. Methods > Evaluation metrics also identifies the evaluators as two radiological technologists and a medical physicist, not radiologists. The Discussion does say these metrics sometimes diverged from human evaluations, but that does not establish the citing sentence's categorical claim. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredStandard NLP metrics e.g. BLEU, ROUGE often fail to capture the nuanced requirements of domains like medicine.
“As previously mentioned, standard NLP metrics e.g. BLEU, ROUGE (Recall-Oriented Understudy for Gisting Evaluation), often fail to capture the nuanced requirements of domains like medicine.”
From Evaluating Metrics for Safety with LLM-as-Judges (2025) · cites this paper as [6]
The Discussion (page not available) says traditional string-based metrics such as ROUGE can undervalue semantically correct medical answers and sometimes show trends opposite to human evaluations. This supports the general point, but the paper does not assess or make that finding about BLEU, the other named example. (LLM, conf=0.84)
Experimental — accuracy not yet measuredIn nuclear medicine QA tasks, LLM-judged metrics only partially correlated with radiologists' assessments.
“For instance, in nuclear medicine QA tasks, LLM-judged metrics only partially correlated with radiologists' assessments, raising concerns about entrusting judgment to models that cannot fully grasp context-specific error severity [6].”
From Evaluating Metrics for Safety with LLM-as-Judges (2025) · cites this paper as [6]
The Results section, “Correlation between human and automated evaluations” (page unavailable), reports weak-to-moderate correlations and limited agreement between LLM-based scores and human ratings. But the Methods section, “Evaluation metrics” (page unavailable), identifies the human evaluators as two radiological technologists and a medical physicist, not radiologists. The paper also does not establish that the disagreement arose from models failing to grasp context-specific error severity. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAutomatic scores align poorly with human expert judgments.
“Research has found that such automatic scores align poorly with human expert judgments [6].”
From Evaluating Metrics for Safety with LLM-as-Judges (2025) · cites this paper as [6]
An LLM-based evaluation metric (RAGAS) achieved more consistent rankings with the experts, but still only modestly correlated to manual scores, implying that expert oversight remains essential.
“An LLM-based evaluation metric (RAGAS) achieved more consistent rankings with the experts, but still only modestly correlated to manual scores, implying that expert oversight remains essential.”
From Evaluating Metrics for Safety with LLM-as-Judges (2025) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 16 · Cited by 9
Research works printed but not linked: 22
- [1]Kaddour J, Harris J, Mozes M, et al. Challenges and Applications of Large Language Models. arXiv. 2023; arXiv:2307.10169. · arXiv
- [2]Chen ZZ, Ma J, Zhang X, et al. A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law. arXiv. 2024; arXiv:2405.01769. · arXiv
- [3]Bruscia M, Manduzio GA, Galatolo FA, et al. An overview on large language models across key domains: a systematic review. In: 2024 IEEE international conference on metrology for eXtended reality, artificial intelligence and neural engineering. 2024. p. 125–30. 10.1109/MetroXRAINE62247.2024.10797032. · DOI
- [4]Chen Z, Hernández Cano A, Romanou A, et al. MEDITRON-70B: scaling medical pretraining for large language models. arXiv. 2023; arXiv:2311.16079. · arXiv
- [5]Saab K, Tu T, Weng W-H, et al. Capabilities of gemini models in medicine. arXiv. 2024; arXiv:2404.18416. · arXiv
- [6]Vrdoljak J, Boban Z, Vilović M, Kumrić M, Božić J. A review of large language models in medical education, clinical decision support, and healthcare administration. Healthcare. 2025;13(6):603. 10.3390/healthcare13060603. 40150453 PMC11942098 · DOI
- [7]Djekidel M. The changing landscape of nuclear medicine and a new era: the “NEW (Nu) CLEAR Medicine”: a framework for the future. Front Nucl Med. 2023. 10.3389/fnume.2023.1213714. 39380958 PMC11460298 · DOI
- [8]Weber WA, Czernin J, Anderson CJ, et al. The future of nuclear medicine, molecular imaging, and theranostics. J Nucl Med. 2020;61(Suppl 2):263S-272S. 10.2967/jnumed.120.254532. 33293447 · DOI
- [9]Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst. 2020;33:9459–74. 10.5555/3495724.3496517. · DOI
- [10]Fan W, Ding Y, Ning L, et al. A survey on RAG meeting LLMs: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. 2024. p. 6491–500. 10.1145/3637528.3671470. · DOI · Paper page
- [11]Miao J, Thongprayoon C, Suppadungsuk S, Valencia OAG, Cheungpasitporn W. Integrating retrieval-augmented generation with large language models in nephrology: advancing practical applications. Medicina. 2024;60(3):445. 10.3390/medicina60030445. 38541171 PMC10972059 · DOI
- [12]Wang D, Liang J, Ye J, et al. Enhancement of the performance of large language models in diabetes education through retrieval-augmented generation: comparative study. J Med Internet Res. 2024;26: e58041. 10.2196/58041. 39046096 PMC11584532 · DOI · Paper page
- [13]Ke Y, Jin L, Elangovan K, et al. Development and testing of retrieval augmented generation in large language models – a case study report. arXiv. 2024; arXiv:2402.01733. · arXiv
- [14]Gumma V, Raghunath A, Jain M, Sitaram S. Health-Pariksha: assessing RAG models for health chatbots in real-world multilingual settings. arXiv. 2024; arXiv:2410.13671. · arXiv
- [15]Kusunose K, Kashima S, Sata M. Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension Guidelines. Circ J. 2023. 10.1253/circj.CJ-23-0308. 37286486 · DOI
- [16]Hisamatsu T, Fukuda M, Kinuta M, Kanda H. ChatGPT responses to clinical questions in the Japan Atherosclerosis Society Guidelines for prevention of atherosclerotic cardiovascular disease 2022. J Atheroscler Thromb. 2024. 10.5551/jat.65240. 39477517 PMC12055503 · DOI
- [17]Sukeda I, Suzuki M, Sakaji H, Kodera S. JMedLoRA: medical domain adaptation on Japanese large language models using instruction-tuning. In: Deep generative models for health workshop, NeurIPS 2023. 2023. https://openreview.net/forum?id=BfHX0hKRSe.
- [21]Guerreiro NM, Alves DM, Waldendorf J. Hallucinations in large multilingual translation models. Trans Assoc Comput Linguist. 2023;11:1500–17. 10.1162/tacl_a_00615. · DOI · Paper page
- [22]Ren R, Qu Y, Liu J, Zhao X. A thorough examination on zero-shot dense retrieval. In: Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. p. 15783–96. 10.18653/v1/2023.findings-emnlp.1057. · DOI
- [25]Kim Y, Jeong H, Chen S, et al. Medical hallucination in foundation models and their impact on healthcare. medRxiv. 2025. 10.1101/2025.02.28.25323115. · DOI
- [26]Gargari OK, Habibi G. Enhancing medical AI with retrieval-augmented generation: a mini narrative review. Digit Health. 2025;11:20552076251337176. 10.1177/20552076251337177. 40343063 PMC12059965 · DOI
- [27]Tomczak M, Tomczak E. The need to report effect size estimates revisited: An overview of some recommended measures of effect size. Trends Sport Sci. 2014;21(1):19–25.
- Authors
- Fukui, Y., Kawata, Y., Kobashi, K., Nagatani, Y., & Iguchi, H.
- Affiliations
- Shiga University of Medical Science (JP)
- Corresponding author
- Fukui, Yusuke
- Venue
- Radiological Physics and Technology
- Published
- 2025-07-19
- Volume
- 18
- Issue
- 3
- Pages
- 861-876
- ISSN
- 1865-0333 (ISSN-L)
- 1865-0341
- PMC
- PMC12339626
- OpenAlex
- W4412520662
- Cited by
- 9
- References
- 16
- Citations by year
- 2026: 8 · 2025: 1
- FWCI
- 2.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Radiological Physics and Technology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- retrieval-augmented generation, hybrid retrieval, GPT-4o, RAGAS evaluation, automated grading, question answering, radiographers
- MeSH terms
- East Asian People, Large Language Models, Automation, Humans, Japan, Language, Nuclear Medicine, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
