Evaluation of the Accuracy of ChatGPT in Answering Clinical Questions on the Japanese Society of Hypertension Guidelines
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
BACKGROUND: To assist healthcare providers in interpreting guidelines, clinical questions (CQ) are often included, but not always, which can make interpretation difficult for non-expert clinicians. We evaluated the ability of ChatGPT to accurately answer CQs on the Japanese Society of Hypertension Guidelines for the Management of Hypertension (JSH 2019). METHODS AND RESULTS: We conducted an observational study using data from JSH 2019. The accuracy rate for CQs and limited evidence-based questions of the guidelines (Qs) were evaluated. ChatGPT demonstrated a higher accuracy rate for CQs than for Qs (80% vs. 36%, P value: 0.005). CONCLUSIONS: ChatGPT has the potential to be a valuable tool for clinicians in the management of hypertension.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Japanese Society of Hypertension Guidelines for the Management of Hypertension (JSH 2019)
Umemura, S., Arima, H., Arima, S., Asayama, K., Dohi, Y., Hirooka, Y., Horio, T., Hoshide, S., Ikeda, S., Ishimitsu, T., et al. (2019) · Hypertension Research · doi:10.1038/s41440-019-0284-9 · Cited by 1,880
Diagnostic error in internal medicine
Graber, M. L., Franklin, N., & Gordon, R. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.13.1493 · Cited by 1,549
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008
Generating scholarly content with ChatGPT: Ethical challenges for medical publishing
Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., & Smith, A. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00019-5 · Cited by 594
Threats by artificial intelligence to human health and human existence
Federspiel, F., Mitchell, R., Asokan, A., Umana, C., & McCoy, D. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2022-010435 · Cited by 156
Cited by
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283
Moulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105474 · Cited by 244
Analyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students
Zawiah, M., Al-Ashwal, F., Gharaibeh, L., Abu Farha, R., Alzoubi, K., Abu Hammour, K., Qasim, Q. A., & Abrah, F. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s439223 · Cited by 51
“22 Another study that was conducted in the US revealed that ChatGPT responded appropriately to 21 of 25 questions (84”
BackgroundFactuality Challenges in the Era of Large Language Models
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.05189 · Cited by 32
Metadata
- Authors
- Kusunose, K., Kashima, S., & Sata, M.
- Venue
- Circulation Journal
- Publisher
- Japanese Circulation Society
- Published
- 2023-06-23
- Volume
- 87
- Issue
- 7
- Pages
- 1030-1033
- ISSN
- 1346-9843 (ISSN-L)
- 1347-4820
- OpenAlex
- W4379511585
- Cited by
- 53
- References
- 5
- FWCI
- 1.83
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.