GPT-4 outperforms ChatGPT in answering non-English questions related to cirrhosis
Open accessAbstract
Background and Objectives: Artificial intelligence is increasingly being employed in healthcare, raising concerns about the exacerbation of disparities. This study evaluates ChatGPT and GPT-4’s ability to comprehend and respond to cirrhosis-related questions in English, Korean, Mandarin, and Spanish, addressing language barriers that may impact patient care. Methods: A set of 36 cirrhosis-related questions were translated into Korean, Mandarin, and Spanish and prompted to both ChatGPT and GPT-4 models. Non-English responses were graded by native-speaking hepatologists on accuracy and similarity to English responses. Chi-square tests were used to compare the proportions of grading between ChatGPT and GPT-4. Results: GPT-4 showed a marked improvement in the proportion of comprehensive and correct answers compared to ChatGPT across all four languages (p<0.05). GPT-4 demonstrated enhanced accuracy and avoided erroneous responses evident in ChatGPT’s output. Significant improvement was observed in Mandarin and Korean subgroups, with a smaller quality gap between English and non-English responses in GPT-4 compared to ChatGPT. Conclusions: GPT-4 exhibited significantly higher accuracy in English and non-English cirrhosis-related questions, highlighting its potential for more accurate and reliable language model applications in diverse linguistic contexts. These advancements have important implications for patients with language discordance, contributing to equalizing health literacy on a global scale.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Yeo et al. highlight that LLMs could provide cross-lingual translation and thus contribute to equalizing healthcare and racial disparities.
“In their work on the use of ChatGPT 4 in overcoming language barriers, Yeo et al. highlight an additional benefit, as LLMs could provide cross-lingual translation and thus contribute to equalizing healthcare and racial disparities56.”
From The ethics of ChatGPT in medicine and healthcare: a systematic review on large language models (LLMs) (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · see citing works
Sepanlou, S. G., Safiri, S., Bisignano, C., Ikuta, K. S., Merat, S., Saberifiroozi, M., Poustchi, H., Tsoi, D., Colombara, D. V., Abdoli, A., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(19)30349-8 · Cited by 1,714 · see citing works
Ginès, P., Krag, A., Abraldes, J. G., Solà, E., Fabrellas, N., & Kamath, P. S. (2021) · The Lancet · doi:10.1016/s0140-6736(21)01374-x · Cited by 1,663 · see citing works
Implications of language barriers for healthcare: a systematic review
Al Shamsi, H., Almutairi, A. G., Al Mashrafi, S., & Al Kalbani, T. (2020) · Oman Medical Journal · doi:10.5001/omj.2020.40 · Cited by 998 · see citing works
Language proficiency and adverse events in US hospitals: a pilot study
Divi, C., Koss, R. G., Schmaltz, S. P., & Loeb, J. M. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzl069 · Cited by 745 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 655 · see citing works
Self-citationSarraju, A., Bruemmer, D., Iterson, E. H. V., Cho, L., Rodríguez, F., & Laffin, L. J. (2023) · JAMA · doi:10.1001/jama.2023.1044 · Cited by 455 · see citing works
Pandey, M., Maina, R. G., Amoyaw, J. A., Li, Y., Kamrul, R., Michaels, C. R., & Maroof, R. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06750-4 · Cited by 383 · see citing works
Policy Shaping: Integrating Human Feedback with Reinforcement Learning
Griffith, S., Subramanian, K. A., Scholz, J., Isbell, C. L., & Thomaz, A. L. (2013) · Cited by 307 · see citing works
Artificial intelligence‐based chatbot patient information on common retinal diseases using ChatGPT
Potapenko, I. O., Boberg‐Ans, L. C., Hansen, M. S., Klefter, O. N., van Dijk, E. H. C., & Subhi, Y. (2023) · Acta Ophthalmologica · doi:10.1111/aos.15661 · Cited by 148 · see citing works
Managing liver cirrhotic complications: Overview of esophageal and gastric varices
Lesmana, C. R. A., Raharjo, M., & Gani, R. A. (2020) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2020.0022 · Cited by 121 · see citing works
Uche-Anya, E. N., Anyane‐Yeboa, A., Berzin, T. M., Ghassemi, M., & May, F. P. (2022) · Gut · doi:10.1136/gutjnl-2021-326271 · Cited by 115 · see citing works
Accuracy of weight loss information in Spanish search engine results on the internet
Cardel, M. I., Chavez, S., Bian, J., Peñaranda, E., Miller, D. R., Huo, T., & Modave, F. P. (2016) · Obesity · doi:10.1002/oby.21646 · Cited by 25 · see citing works
Readability and Quality of English and Spanish Online Health Information about Cochlear Implants
Nix, E., Willgruber, A., Rawls, C., Kinealy, B. P., Zeitler, D. M., Schuh, M. R., & Bush, M. L. (2022) · Otology & Neurotology · doi:10.1097/mao.0000000000003791 · Cited by 16 · see citing works
Garland, M. E., Lukac, D., & Contreras, P. (2022) · Journal of Cancer Education · doi:10.1007/s13187-022-02171-x · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Wenxiang Jiao WW, Jen-tse Huang, Xing Wang, Zhaopeng Tu. Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine. arXiv 2023.
- Linting Xue NC, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. Conference of the North American Chapter of the Association for . Computational Linguistics: Human Language Technologies, NAACL-HLT 2021: Association for Computational Linguistics, 2021.
Cited by
Sorted by citations
Haltaufderheide, J., & Ranisch, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01157-x · Cited by 396 · see citing works
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
ChatGPT failed Taiwan’s Family Medicine Board Exam
Weng, T.-L., Wang, Y.-M., Chang, S., Chen, T.-J., & Hwang, S.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000946 · Cited by 100 · see citing works
Ha, J., Jeon, H., Han, D. S., Seo, J., & Oh, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642472 · Cited by 76 · see citing works
Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis
Yu, H., Fan, L., Li, L., Zhou, J., Ma, Z., Xian, L., Hua, W., He, S., Jin, M., Zhang, Y., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00171-8 · Cited by 69 · see citing works
Self-citationChatGPT: Transforming Healthcare with AI
Neha, F., Bhati, D., Shukla, D. K., & Amiruzzaman, M. (2024) · AI · doi:10.3390/ai5040126 · Cited by 47 · see citing works
Appropriateness of ChatGPT in Answering Heart Failure Related Questions
King, R. C., Samaan, J. S., Yeo, Y. H., Mody, B. P., Lombardo, D. M., & Ghashghaei, R. (2024) · Heart Lung and Circulation · doi:10.1016/j.hlc.2024.03.005 · Cited by 45 · see citing works
Self-citationNedbal, C., Naik, N., Castellani, D., Gauhar, V., Geraghty, R. M., & Somani, B. K. (2023) · Current Opinion in Urology · doi:10.1097/mou.0000000000001151 · Cited by 38 · see citing works
Large Language Models in Gastroenterology: Systematic Review
Gong, E. J., Bang, C. S., Lee, J. J., Park, J., Kim, E., Kim, S., Kimm, M., & Choi, S.-H. (2024) · Journal of Medical Internet Research · doi:10.2196/66648 · Cited by 41 · see citing works
Sallam, M., Al-Mahzoum, K., Alshuaib, O., Alhajri, H., Alotaibi, F., Alkhurainej, D., Al-Balwah, M. Y., Barakat, M. M., & Egger, J. (2024) · BMC Infectious Diseases · doi:10.1186/s12879-024-09725-y · Cited by 36 · see citing works
Ando, K., Masaki, S., Wakatsuki, S., Nagai, R., Chino, K., Kai, H., Sasaki, T., Kato, R., Nguyen, T. P., Guo, N., et al. (2024) · BJA Open · doi:10.1016/j.bjao.2024.100296 · Cited by 31 · see citing works
Waters, M. R., Aneja, S., & Hong, J. C. (2023) · Practical Radiation Oncology · doi:10.1016/j.prro.2023.06.011 · Cited by 21 · see citing works
Expanding horizons and navigating challenges for enhanced clinical workflows: ChatGPT in urology
Talyshinskii, A., Naik, N., Hameed, B. M. Z., Zhanbyrbekuly, U., Khairli, G., Гулиев, Б. Г., Juilebø-Jones, P., Tzelves, L. I., & Somani, B. K. (2023) · Frontiers in Surgery · doi:10.3389/fsurg.2023.1257191 · Cited by 20 · see citing works
Ye, F., Zhang, H., Luo, X., Wu, T., Yang, Q., & Shi, Z. (2024) · Otolaryngology · doi:10.1002/ohn.832 · Cited by 19 · see citing works
Triantafyllopoulos, L., Feretzakis, G., Tzelves, L. I., Sakagianni, A., Verykios, V. S., & Kalles, D. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108404 · Cited by 20 · see citing works
Yang, C. T.-Y., & Chen, H. H.-J. (2025) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2453205 · Cited by 19 · see citing works
A Multidisciplinary Assessment of ChatGPT’s Knowledge of Amyloidosis: Observational Study
King, R. C., Samaan, J. S., Yeo, Y. H., Peng, Y., Kunkel, D. C., Habib, A. A., & Ghashghaei, R. (2024) · JMIR Cardio · doi:10.2196/53421 · Cited by 17 · see citing works
Self-citationSamaan, J. S., Rajeev, N. D., Ng, W. H., Srinivasan, N., Busam, J. A., Yeo, Y. H., & Samakar, K. (2024) · Obesity Surgery · doi:10.1007/s11695-024-07212-6 · Cited by 15 · see citing works
Self-citationSamaan, J. S., Issokson, K., Feldman, E., Fasulo, C., Rajeev, N. D., Ng, W. H., Hollander, B. S., Yeo, Y. H., & Vasiliauskas, E. A. (2024) · Crohn s & Colitis 360 · doi:10.1093/crocol/otae077 · Cited by 14 · see citing works
Self-citationAjit‐Roger, E., Moise, A., Peralta, C., Orishchak, O., & Daniel, S. J. (2024) · OTO Open · doi:10.1002/oto2.70048 · Cited by 11 · see citing works
Gong, E. J., Bang, C. S., Lee, J. J., Park, J., Kim, E., Kim, S., Kimm, M., & Choi, S.-H. (2024) · Bioengineering · doi:10.3390/bioengineering12010001 · Cited by 11 · see citing works
Yoon, S.-H., Oh, S. K., Lim, B. G., & Lee, H.-J. (2024) · JMIR Medical Education · doi:10.2196/56859 · Cited by 10 · see citing works
Real-World Performance of Large Language Models in Emergency Department Chest Pain TriagePreprint
Meng, X., Ji, J., Yan, X., Xu, H., Gao, J., Wang, J., Wang, J., Wang, X., Wang, Y.-G.-S., Wang, W., et al. (2024) · medRxiv · doi:10.1101/2024.04.24.24306264 · Cited by 7 · see citing works
Yeo, Y. H., Samaan, J. S., & Ng, W. H. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0183 · Cited by 5 · see citing works
Self-citationAssessing the ChatGPT aptitude: A competent and effective Dermatology doctor?
Lian, C., Yuan, X., Chokkakula, S., Wang, G., Song, B., Wang, Z., Fan, G., & Yin, C. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e37220 · Cited by 4 · see citing works
Evaluating Large Language Models in Ptosis-Related inquiries: A Cross-Lingual Study
Niu, L.-H., Li, W., Qin, B., Chen, T., Dong, L., He, Y., Jiang, X., Wang, M., Ma, L., Geng, J., et al. (2025) · Translational Vision Science & Technology · doi:10.1167/tvst.14.7.9 · Cited by 3 · see citing works
Moise, A., Tatar, L., Sela, N., da Silva, S. D., Kouz, J., Tamilia, M., Hier, M. P., Forest, V., & Payne, R. J. (2025) · Journal of Otolaryngology - Head and Neck Surgery · doi:10.1177/19160216251387617 · Cited by 3 · see citing works
Towards Responsible AI in Patient Education: Ethical, Linguistic, and Regulatory Considerations
Samaan, J. S., Yeo, Y. H., Rajeev, N. D., Ng, W. H., Srinivasan, N., & Samakar, K. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06877-9 · Cited by 3 · see citing works
Self-citationInclusive AI in Healthcare: Enhancing Bariatric Surgery Education for Diverse Patient Populations
Samaan, J. S., Yeo, Y. H., Rajeev, N. D., Ng, W. H., Srinivasan, N., & Samakar, K. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06969-6 · Cited by 2 · see citing works
Self-citationSchlussel, L., Samaan, J. S., Chan, Y. Y., Chang, B. W., Yeo, Y. H., Ng, W. H., & Rezaie, A. R. (2024) · Artificial Intelligence in Gastroenterology · doi:10.35712/aig.v5.i1.90503 · Cited by 1 · see citing works
Self-citationEvaluating ChatGPT 4.0’s User Satisfaction Among Doctors Across Different Medical Departments
Triantafyllopoulos, L., Feretzakis, G., Tzelves, L. I., Sakagianni, A., Verykios, V. S., & Kalles, D. (2024) · Studies in health technology and informatics · doi:10.3233/shti240469 · Cited by 1 · see citing works
Ferrazzani, S., Cornélissen, G., Caruso, A., Carolis, S. D., Hillman, D., Giglio, R. D., Martino, I. V., Pometti, A., Tocci, A., & Kato, J. (1990) · PubMed · Cited by 1 · see citing works
A multidisciplinary assessment of ChatGPT’s knowledge of amyloidosisPreprint
King, R. C., Samaan, J. S., Yeo, Y. H., Kunkel, D. C., Habib, A. A., & Ghashghaei, R. (2023) · medRxiv · doi:10.1101/2023.07.17.23292780
Self-citationTang, Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4785726/v1
GPT-4 Enhances Readability of Online Patient Materials for Cardiac AmyloidosisPreprint
Haquang, J., Bharani, V., Samaan, J. S., King, R. C., Margolis, S., Srinivasan, N., Rajeev, N. D., Chan, J., Yeo, Y. H., & Ghashghaei, R. (2025) · medRxiv · doi:10.1101/2025.05.19.25325053
Self-citation
- Authors
- Yeo, Y. H., Samaan, J. S., Ng, W. H., Ma, X., Ting, P.-S., Kwak, M.-S., Panduro, A., Lizaola-Mayo, B., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., & Kuo, A.
- Affiliations
- Cedars-Sinai Medical Center (US)
- University of Bristol (GB)
- Nanjing Drum Tower Hospital (CN)
- Tulane University (US)
- Seoul National University (KR)
- Universidad de Guadalajara (MX)
- Corresponding author
- Spiegel, Brennan
- Kuo, Alexander
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-05-05
- OpenAlex
- W4372348574
- Cited by
- 35
- References
- 15
- Citations by year
- 2025: 5 · 2024: 21 · 2023: 8
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Interpreting and Communication in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- GPT-4, ChatGPT, liver cirrhosis, language barriers, multilingual question answering, health disparities, health literacy, language discordance, large language models in healthcare, Korean language, mandarin, Spanish language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
