Professionalism and clinical short answer question marking with machine learning
Abstract
Machine learning may assist in medical student evaluation. This study involved scoring short answer questions administered at three centres. Bidirectional encoder representations from transformers were particularly effective for professionalism question scoring (accuracy ranging from 41.6% to 92.5%). In the scoring of 3‐mark professionalism questions, as compared with clinical questions, machine learning had a lower classification accuracy ( P < 0.05). The role of machine learning in medical professionalism evaluation warrants further investigation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI was used to grade exams, particularly those involving short-answer, long-answer, and essay-based responses.
“AI was also used to grade exams, particularly those that involve short-answer, long-answer, and essay-based responses [137–139].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [139]
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
Natural language processing with Python
Bird, S., Klein, E., & Loper, E. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,472 · see citing works
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886 · see citing works
(1966) · Phi Delta Kappan · Cited by 449 · see citing works
Artificial Intelligence in Education
Holmes, W. (2019) · Encyclopedia of Education and Information Technologies · doi:10.1007/978-3-319-60013-0_107-1 · Cited by 336 · see citing works
Machine learning and medical education
Kolachalama, V. B., & Garg, P. S. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0061-1 · Cited by 320 · see citing works
Using Machine Learning to Assess Physician Competence: A Systematic Review
DIAS, R. D., Gupta, A., & Yule, S. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002414 · Cited by 70 · see citing works
Peer assessment in the objective structured clinical examination: A scoping review
Khan, R., Payne, M. W. C., & Chahine, S. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1309375 · Cited by 63 · see citing works
Assessing Professionalism in Medicine – A Scoping Review of Assessment Tools from 1990 to 2018
Tay, K. T., Ng, S., Hee, J. M., Chia, E. W. Y., Vythilingam, D. I., Ong, Y. T., Chiam, M., Chin, A., Fong, W. W. S., Wijaya, L., et al. (2020) · Journal of Medical Education and Curricular Development · doi:10.1177/2382120520955159 · Cited by 55 · see citing works
Sukkarieh, J. Z., & Pulman, S. (2005) · Cited by 38 · see citing works
Tiffin, P. A., Finn, G. M., & McLachlan, J. C. (2011) · BMC Medical Education · doi:10.1186/1472-6920-11-43 · Cited by 27 · see citing works
Impact of panelists’ experience on script concordance test scores of medical students
Peyrony, O., Hutin, A., Truchot, J., Borie, R., Calvet, D., Albaladejo, A., Baadj, Y., Cailleaux, P., Flamant, M., Martin, C., et al. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02243-w · Cited by 24 · see citing works
A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models
Mokrii, I., Boytsov, L., & Braslavski, P. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463093 · Cited by 24 · see citing works
Automated Essay Scoring Using Transformer Models
Ludwig, S., Mayer, C., Hansen, C., Eilers, K., & Brandt, S. (2021) · Psych · doi:10.3390/psych3040056 · Cited by 22 · see citing works
Blacketer, C., Parnis, R., B. Franke, K., Wagner, M., Wang, D., Tan, Y., Oakden‐Rayner, L., Gallagher, S., Perry, S. W., Licinio, J., et al. (2021) · Internal Medicine Journal · doi:10.1111/imj.15479 · Cited by 17 · see citing works
Self-citationAnalysis of Algorithms in Automated Marking in Education: A Proposed Hybrid Algorithm
Prasain, B., & Bajaj, S. K. (2020) · doi:10.1109/citisia50690.2020.9371783 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
AI in Medical Education: Global situation, effects and challenges
Zhang, W., Cai, M., Lee, H. J., Evans, R., Zhu, C., & Ming, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12009-8 · Cited by 148 · see citing works
Artificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
“AI was also used to grade exams, particularly those that involve short-answer, long-answer, and essay-based responses [137–139].”
- Authors
- Lam, A., Lam, L., Blacketer, C., Parnis, R., Franke, K., Wagner, M., Wang, D., Tan, Y., Oakden‐Rayner, L., Gallagher, S., Perry, S. W., Licinio, J., Symonds, I., Thomas, J., Duggan, P., & Bacchi, S.
- Affiliations
- The University of Adelaide (AU)
- Royal Adelaide Hospital (AU)
- Royal Darwin Hospital (AU)
- SUNY Upstate Medical University (US)
- University of Otago (NZ)
- State University of New York (SUNY) Upstate Medical University Syracuse New York USA
- Corresponding author
- Bacchi, Stephen
- ORCID
- Venue
- Internal Medicine Journal
- Publisher
- Wiley
- Published
- 2022-07
- Volume
- 52
- Issue
- 7
- Pages
- 1268-1271
- ISSN
- 1444-0903 (ISSN-L)
- 1445-5994
- OpenAlex
- W4287845110
- Cited by
- 8
- References
- 19
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 2 · 2023: 1
- FWCI
- 1.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Internal Medicine Journal (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- professionalism assessment, short-answer questions, automated grading, medical student assessment, BERT, medical education assessment
- MeSH terms
- Professionalism, Machine Learning, Humans, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
