Trialling a Large Language Model (ChatGPT) in General Practice With the Applied Knowledge Test: Observational Study Demonstrating Opportunities and Limitations in Primary Care
Open accessPublished version of a preprint
Abstract
Background: Large language models exhibiting human-level performance in specialized tasks are emerging; examples include Generative Pretrained Transformer 3.5, which underlies the processing of ChatGPT. Rigorous trials are required to understand the capabilities of emerging technology, so that innovation can be directed to benefit patients and practitioners. Objective: Here, we evaluated the strengths and weaknesses of ChatGPT in primary care using the Membership of the Royal College of General Practitioners Applied Knowledge Test (AKT) as a medium. Methods: AKT questions were sourced from a web-based question bank and 2 AKT practice papers. In total, 674 unique AKT questions were inputted to ChatGPT, with the model’s answers recorded and compared to correct answers provided by the Royal College of General Practitioners. Each question was inputted twice in separate ChatGPT sessions, with answers on repeated trials compared to gauge consistency. Subject difficulty was gauged by referring to examiners’ reports from 2018 to 2022. Novel explanations from ChatGPT—defined as information provided that was not inputted within the question or multiple answer choices—were recorded. Performance was analyzed with respect to subject, difficulty, question source, and novel model outputs to explore ChatGPT’s strengths and weaknesses. Results: Average overall performance of ChatGPT was 60.17%, which is below the mean passing mark in the last 2 years (70.42%). Accuracy differed between sources (P=.04 and .06). ChatGPT’s performance varied with subject category (P=.02 and .02), but variation did not correlate with difficulty (Spearman ρ=–0.241 and –0.238; P=.19 and .20). The proclivity of ChatGPT to provide novel explanations did not affect accuracy (P>.99 and .23). Conclusions: Large language models are approaching human expert–level performance, although further development is required to match the performance of qualified primary care physicians in the AKT. Validated high-performance models may serve as assistants or autonomous clinical tools to ameliorate the general practice workforce crisis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT's average score in the student comprehensive assessment in primary care falls below the passing score.
“Thirunavukarasu et al. [186] assessed ChatGPT's performance in primary care and found that its average score in the student comprehensive assessment falls below the passing score, indicating room for improvement.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [186]
The Results report ChatGPT’s average score as 60.17%, below the recent mean AKT passing mark of 70.42%. But the Introduction identifies the AKT as an examination UK GPs must pass to complete their training, not a student comprehensive assessment; the claim misidentifies the assessment. (LLM, conf=0.88)
Experimental — accuracy not yet measuredEven in English, in a real clinical setting, GPT-3.5 cannot answer questions at a level acceptable to fully qualified primary care physicians.
“A previous study suggests that even in English, in a real clinical setting, GPT-3.5 cannot answer questions at a level acceptable to fully qualified primary care physicians [10].”
From Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study (2023) · cites this paper as [10]
When using GPT-3.5 or GPT-4 for medical applications, their differing accuracy across specialties must be taken into account.
“However, when using either GPT-3.5 or GPT-4 for medical applications, its differing accuracy across specialties must also be taken into account [48].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Addressing the crisis of GP recruitment and retention: a systematic review
Marchand, C., & Peckham, S. (2017) · British Journal of General Practice · doi:10.3399/bjgp17x689929 · Cited by 238 · see citing works
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199 · see citing works
Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study
Nov, O., Singh, N., & Mann, D. (2023) · JMIR Medical Education · doi:10.2196/46939 · Cited by 198 · see citing works
Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022) · doi:10.1101/2022.12.23.22283901 · Cited by 172 · see citing works
Tossaint-Schoenmakers, R., Versluis, A., Chavannes, N. H., Talboom-Kamp, E. P. W. A., & Kasteleyn, M. J. (2021) · Journal of Medical Internet Research · doi:10.2196/27180 · Cited by 156 · see citing works
Digital health during COVID-19: lessons from operationalising new models of care in ophthalmology
Gunasekeran, D. V., Tham, Y. C., Ting, D. S. W., Tan, G. S. W., & Wong, T. Y. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30287-9 · Cited by 151 · see citing works
General Practitioners’ Attitudes Toward Artificial Intelligence–Enabled Systems: Interview Study
Buck, C., Doctor, E., Hennrich, J., Jöhnk, J., & Eymann, T. (2022) · Journal of Medical Internet Research · doi:10.2196/28916 · Cited by 147 · see citing works
Health-focused conversational agents in person-centered care: a review of apps
Parmar, P., Ryu, J., Pandya, S., Sedoc, J., & Agarwal, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00560-6 · Cited by 113 · see citing works
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106 · see citing works
Nussbaum, C. A., Massou, E., Fisher, R. F., Morciano, M., Harmer, R., & Ford, J. A. (2021) · BJGP Open · doi:10.3399/bjgpo.2021.0066 · Cited by 99 · see citing works
Shortage of general practitioners in the NHS
Majeed, A. (2017) · BMJ · doi:10.1136/bmj.j3191 · Cited by 81 · see citing works
Ford, E., Oswald, M., Hassan, L., Bozentko, K., Nenadic, G., & Cassell, J. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105472 · Cited by 59 · see citing works
Machine Learning: The Next Paradigm Shift in Medical Education
James, C. A., Wheelock, K. M., & Woolliscroft, J. O. (2021) · Academic Medicine · doi:10.1097/acm.0000000000003943 · Cited by 58 · see citing works
Putting ChatGPT’s Medical Advice to the (Turing) TestPreprint
Nov, O., Singh, N. P., & Mann, D. M. (2023) · medRxiv · doi:10.1101/2023.01.23.23284735 · Cited by 50 · see citing works
Fletcher, E., Burns, A., Wiering, B., Lavu, D., Shephard, E., Hamilton, W., Campbell, J. L., & Abel, G. (2023) · BMC Primary Care · doi:10.1186/s12875-023-01973-2 · Cited by 40 · see citing works
Razai, M. S., & Majeed, A. (2022) · Journal of Ambulatory Care Management · doi:10.1097/jac.0000000000000410 · Cited by 36 · see citing works
Examining Mental Workload Relating to Digital Health Technologies in Health Care: Systematic Review
Kremer, L., Lipprandt, M., Röhrig, R., & Breil, B. (2022) · Journal of Medical Internet Research · doi:10.2196/40946 · Cited by 29 · see citing works
Physicians’ Perceptions of the Use of a Chatbot for Information Seeking: Qualitative Study
Koman, J., Fauvelle, K., Schück, S., Texier, N., & Mebarki, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15185 · Cited by 21 · see citing works
Upon Improving the Performance of Localized Healthcare Virtual Assistants
Malamas, N., Papangelou, K., & Symeonidis, A. L. (2022) · Healthcare · doi:10.3390/healthcare10010099 · Cited by 19 · see citing works
Thirunavukarasu, A. J., Hassan, R., Limonard, A., & Savant, S. V. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281847 · Cited by 17 · see citing works
Self-citationGeneral practice work and workforce: Interdependencies between demand, supply and quality
Sturmberg, J. P., O’Halloran, D., McDonnell, G., et al. (2018) · Australian Journal of General Practice · doi:10.31128/ajgp-03-18-4515 · Cited by 13 · see citing works
Thirunavukarasu, A. J., Hassan, R., Limonard, A., & Savant, S. V. (2023) · medRxiv · doi:10.1101/2023.02.03.23285417 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [2]2. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler D, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D. Language models are few-shot learners. 34th Conference on Neural Information Processing Systems (NeurIPS 2020); December 6-12, 2020; Vancouver, BC. 2020. https://papers.nips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
Cited by
Sorted by citations
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Self-citationA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · see citing works
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
Large language models in health care: Development, applications, and challenges
Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., & Liu, N. (2023) · Health Care Science · doi:10.1002/hcs2.61 · Cited by 366 · see citing works
Self-citationRoles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review (Preprint)
Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024) · doi:10.2196/preprints.56930 · Cited by 302 · see citing works
Artificial intelligence in surgery
Varghese, C., Harrison, E. M., O’Grady, G., & Topol, E. J. (2024) · Nature Medicine · doi:10.1038/s41591-024-02970-3 · Cited by 287 · see citing works
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · see citing works
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234 · see citing works
“The availability of ChatGPT as an example of LLMs carries transformative societal implications, especially in health care settings, making its adoption in health care education seemingly inevitable [9,11,51-54].”
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I. (2024) · NEJM AI · doi:10.1056/aidbp2300192 · Cited by 135 · see citing works
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128 · see citing works
Ng, F. Y. C., Thirunavukarasu, A. J., Cheng, H., Tan, T. F., Gutierrez, L., Lan, Y., Ong, J. C. L., Chong, Y. S., Ngiam, K. Y., Ho, D., et al. (2023) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101230 · Cited by 113 · see citing works
Self-citationMeyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110 · see citing works
Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) · JMIR Medical Education · doi:10.2196/50658 · Cited by 99 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Reasoning with large language models for medical question answering
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae131 · Cited by 66 · see citing works
Large language models will not replace healthcare professionals: curbing popular fears and hype.
Thirunavukarasu, A. J. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231173123 · Cited by 63 · see citing works
Self-citationYousefi, F., Dehnavieh, R., Laberge, M., Gagnon, M.-P., Ghaemi, M. M., Nadali, M., & Azizi, N. (2025) · BMC Primary Care · doi:10.1186/s12875-025-02785-2 · Cited by 48 · see citing works
Cherrez-Ojeda, I., Gallardo-Bastidas, J. C., Robles-Velasco, K., Osorio, M. F., Velez Leon, E. M., Leon Velastegui, M., Pauletto, P., Aguilar-Díaz, F. C., Squassi, A., González Eras, S. P., et al. (2024) · JMIR Medical Education · doi:10.2196/51757 · Cited by 46 · see citing works
Liu, F., Chang, X., Zhu, Q., Huang, Y., Li, Y., & Wang, H. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06232-1 · Cited by 35 · see citing works
- Authors
- Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S.
- Affiliations
- University of Cambridge (GB)
- Corresponding author
- Thirunavukarasu, Arun James
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-04-21
- Volume
- 9
- Pages
- e46599
- ISSN
- 2369-3762 (ISSN-L)
- OpenAlex
- W4364378939
- Cited by
- 187
- References
- 29
- Citations by year
- 2026: 22 · 2025: 53 · 2024: 75 · 2023: 37
- FWCI
- 7.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- ChatGPT, large language models, general practice, primary care, applied knowledge test, Royal College of General Practitioners, medical education, AI performance evaluation, question difficulty, workforce crisis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
