The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today
Open accessAbstract
Recent investigations show that large language models (LLMs), specifically GPT-4, not only have remarkable capabilities in common Natural Language Processing (NLP) tasks but also exhibit human-level performance on various professional and academic benchmarks. However, whether GPT-4 can be directly used in practical applications and replace traditional artificial intelligence (AI) tools in specialized domains requires further experimental validation. In this paper, we explore the potential of LLMs such as GPT-4 to outperform traditional AI tools in dementia diagnosis. Comprehensive comparisons between GPT-4 and traditional AI tools are conducted to examine their diagnostic accuracy in a clinical setting. Experimental results on two real clinical datasets show that, although LLMs like GPT-4 demonstrate potential for future advancements in dementia diagnosis, they currently do not surpass the performance of traditional AI tools. The interpretability and faithfulness of GPT-4 are also evaluated by comparison with real doctors. We discuss the limitations of GPT-4 in its current state and propose future research directions to enhance GPT-4 in dementia diagnosis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In the field of medical assistance, LLMs demonstrate potential applications, including research on identifying gastrointestinal diseases [99], dementia diagnosis [217], accelerating the evaluation of COVID-19 literature [93], and their overall potential in healthcare [17].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [217]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Flagged first, then by citations
Research works printed but not linked: 31
- Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5-32.
- Leo Breiman. 2017. Classification and regression trees. Routledge, Routledge.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
- Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). · arXiv
- Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785-794.
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017). · arXiv
- Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2023. A Bibliometric Review of Large Language Models Research from 2017 to 2023. arXiv preprint arXiv:2304.02020 (2023). · arXiv
- Yongbin He, Haifeng Tang, Dongxue Wang, Shuqin Gu, Guoxin Ni, and Haiyang Wu. 2023. Will ChatGPT/GPT-4 be a lighthouse to guide spinal surgeons? Annals of Biomedical Engineering (2023), 1-4.
- Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. 2021. Training data leakage analysis in language models. arXiv preprint arXiv:2101.05405 (2021). · arXiv
- Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421.
- Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019). · arXiv
- Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. Evaluat- ing gpt-4 and chatgpt on japanese medical licensing examinations. arXiv preprint arXiv:2303.18027 (2023). · arXiv
- David G Kleinbaum, K Dietz, M Gail, Mitchel Klein, and Mitchell Klein. 2002. Logistic regression. Springer, Springer.
- Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS digital health 2, 2 (2023), e0000198.
- Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of GPT-4 as an AI Chatbot for medicine. New England Journal of Medicine 388, 13 (2023), 1233-1239.
- Zachary C Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31-57.
- Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 (2023). · arXiv
- Javier Olazarán, Barry Reisberg, Linda Clare, Isabel Cruz, Jordi Peña-Casanova, Teodoro Del Ser, Bob Woods, Cornelia Beck, Stefanie Auer, Claudia Lai, et al. 2010. Nonpharmacological therapies in Alzheimer's disease: a systematic review of efficacy. Dementia and geriatric cognitive disorders 30, 2 (2010), 161-178.
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs. · arXiv
- Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A large- scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning. PMLR, 248-260.
- Sebastian Palmqvist, Pontus Tideman, Nicholas Cullen, Henrik Zetterberg, Kaj Blennow, Jeffery L Dage, Erik Stomrud, Shorena Janelidze, Niklas Mattsson-Carlgren, and Oskar Hansson. 2021. Prediction of future Alzheimer's disease dementia using plasma phospho-tau combined with other accessible measures. Nature Medicine 27, 6 (2021), 1034-1042.
- Peter S. Park, Philipp Schoenegger, and Chongyang Zhu. 2023. "Correct answers" from the psychol- ogy of artificial intelligence. arXiv:2302.07267 [cs. · arXiv
- Roy H Perlis. 2023. Application of GPT-4 to select next-step antidepressant treatment in major depression. medRxiv (2023), 2023-04.
- Ronald Carl Petersen, PS Aisen, Laurel A Beckett, MC Donohue, AC Gamst, Danielle J Harvey, CR Jack, WJ Jagust, LM Shaw, AW Toga, et al. 2010. Alzheimer's disease neuroimaging initiative (ADNI): clinical characterization. Neurology 74, 3 (2010), 201-209.
- N Saravanan, G Sathish, and JM Balajee. 2018. Data wrangling and data leakage in machine learning for healthcare. International Journal of Emerging Technologies and Innovative Research 5, 8 (2018), 553-557.
- Saeid Ashraf Vaghefi, Qian Wang, Veruska Muccione, Jingwei Ni, Mathias Kraus, Julia Bingler, Tobias Schimanski, Chiara Colesanti-Senni, Nicolas Webersinke, Christrian Huggel, et al. 2023. chatIPCC: Grounding Conversational AI in Climate Science. arXiv preprint arXiv:2304.05510 (2023). · arXiv
- Maxim Vidgof, Stefan Bachhofner, and Jan Mendling. 2023. Large Language Models for Business Process Management: Opportunities and Challenges. arXiv:2304.04309 [cs. · arXiv
- Jie Wang, Zhuo Wang, Ning Liu, Caiyan Liu, Chenhui Mao, Liling Dong, Jie Li, Xinying Huang, Dan Lei, Shanshan Chu, et al. 2022b. Random Forest Model in the Diagnosis of Dementia Patients with Normal Mini-Mental State Examination Scores. Journal of personalized medicine 12, 1 (2022), 37.
- Zhuo Wang, Jie Wang, Ning Liu, Caiyan Liu, Xiuxing Li, Liling Dong, Rui Zhang, Chenhui Mao, Zhichao Duan, Wei Zhang, et al. 2022a. Learning Cognitive-Test-Based Interpretable Rules for Prediction and Early Diagnosis of Dementia Using Neural Networks. Journal of Alzheimer's disease: JAD 90, 2 (2022), 609-624.
- Zhuo Wang, Wei Zhang, Ning Liu, and Jianyong Wang. 2021. Scalable Rule-Based Representa- tion Learning for Interpretable Classification. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A Survey of Large Language Models. arXiv preprint arXiv:2303.18223 (2023). · arXiv
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Self-citationHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Williams, C. Y. K., Miao, B. Y., Kornblith, A. E., & Butte, A. J. (2024) · Nature Communications · doi:10.1038/s41467-024-52415-1 · Cited by 119
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Metadata
- Authors
- Wang, Z., Li, R., Dong, B., Wang, J., Li, X., Liu, N., Mao, C., Zhang, W., Dong, L., Gao, J., & Wang, J.
- Affiliations
- Tsinghua University (CN)
- Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
- First Affiliated Hospital of Nanchang University (CN)
- Chinese Academy of Sciences (CN)
- Shandong University (CN)
- East China Normal University (CN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.01499
- OpenAlex
- W4379474412
- Cited by
- 26
- References
- 1
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 11 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. Department of Defense (U01 AG024904, W81XWH, W81XWH-12-2-, W81XWH1220012, AG024904)
- Alzheimer's Association (U01 AG024904, W81XWH-12-2-0012)
- Bristol-Myers Squibb (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Eli Lilly and Company (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Pfizer (AG024904, W81XWH-12-2-0012, U01 AG024904)
- Biogen (W81XWH-12-2-0012, U01 AG024904, AG024904)
- University of Southern California (W81XWH-12-2-0012, U01 AG024904)
- Meso Scale Diagnostics (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Alzheimer's Disease Neuroimaging Initiative (W81XWH-12-2–0012, U01AG024904, AG024904, W81XWH-12-2-)
- BioClinica (U01 AG024904, W81XWH-12-2-0012, AG024904)
- Novartis Pharmaceuticals Corporation (W81XWH-12-2-0012, U01 AG024904, AG024904)
- Northern California Institute for Research and Education (U01 AG024904, W81XWH-12-2-0012)
- National Natural Science Foundation of China (U01 AG024904, 62272264, 61521002, W81XWH-12-2-0012)
- Eisai (W81XWH-12-2-0012, U01 AG024904, AG024904)
- Servier (AG024904, W81XWH-12-2-0012, U01 AG024904)
- H. Lundbeck A/S (U01 AG024904, AG024904, W81XWH-12-2-0012)
- IXICO (AG024904, W81XWH-12-2-0012, U01 AG024904)
- National Institutes of Health (3u01ag024904-10s2, W81XWH, W81XWH-12-2-0012, U01-AG024904, AG024904)
- Genentech (W81XWH-12-2-0012, AG024904, U01 AG024904)
- Canadian Institutes of Health Research (U01 AG024904, W81XWH-12-2-0012)
- National Key Research and Development Program of China (U01 AG024904, 2020YFA0804501, 2020YFA0804503, W81XWH-12-2-0012)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- GPT-4, large language models, dementia diagnosis, clinical decision support, diagnostic accuracy, interpretability, faithfulness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
