Automated essay scoring and the future of educational assessment in medical education
Abstract
CONTEXT: Constructed-response tasks, which range from short-answer tests to essay questions, are included in assessments of medical knowledge because they allow educators to measure students' ability to think, reason, solve complex problems, communicate and collaborate through their use of writing. However, constructed-response tasks are also costly to administer and challenging to score because they rely on human raters. One alternative to the manual scoring process is to integrate computer technology with writing assessment. The process of scoring written responses using computer programs is known as 'automated essay scoring' (AES). METHODS: An AES system uses a computer program that builds a scoring model by extracting linguistic features from a constructed-response prompt that has been pre-scored by human raters and then, using machine learning algorithms, maps the linguistic features to the human scores so that the computer can be used to classify (i.e. score or grade) the responses of a new group of students. The accuracy of the score classification can be evaluated using different measures of agreement. RESULTS: Automated essay scoring provides a method for scoring constructed-response tests that complements the current use of selected-response testing in medical education. The method can serve medical educators by providing the summative scores required for high-stakes testing. It can also serve medical students by providing them with detailed feedback as part of a formative assessment process. CONCLUSIONS: Automated essay scoring systems yield scores that consistently agree with those of human raters at a level as high, if not higher, as the level of agreement among human raters themselves. The system offers medical educators many benefits for scoring constructed-response tasks, such as improving the consistency of scoring, reducing the time required for scoring and reporting, minimising the costs of scoring, and providing students with immediate feedback on constructed-response tasks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GPT-3 could be used to grade exams or quizzes more quickly and accurately, allowing for more timely feedback to students.
“GPT-3 could also be used to grade exams or quizzes more quickly and accurately, allowing for more timely feedback to students (Gierl et al., 2014).”
From Chatting and cheating: Ensuring academic integrity in the era of ChatGPT (2024) · cites this paper as Gierl et al., 2014
Gierl et al. (2014) use of LightSIDE to grade postgraduate medical student essays resulted in an agreement between computer classification and human raters between 94.6% and 98.2%.
“Gierl, Latifi, Lai, Boulais, and Champlain's (2014) use of open source Java software LightSIDE to grade postgraduate medical student essays resulted in an agreement between the computer classification and human raters between 94.6% and 98.2%”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Gierl, Latifi, Lai, Boulais, and Champlain's (2014)
Automated essay scoring scans the entire document and can understand the overall meaning and the concept even if written with grammatical errors.
“AES scans the entire document and can understand the overall meaning and the concept even if written with grammatical errors [26].”
From Artificial Intelligence Revolutionizing the Field of Medical Education (2023) · cites this paper as [26]
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Assessment and Classroom Learning
Black, P., & Wiliam, D. (1998) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/0969595980050102 · Cited by 8,152 · cited 1× · see citing works
“23 development in which these writing-response tasks are scored using the algorithmic power of modern computers.”
(2005) · PubMed · Cited by 7,011 · see citing works
Kluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882 · see citing works
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631 · see citing works
The second machine age: work, progress, and prosperity in a time of brilliant technologies
(2015) · Choice Reviews Online · doi:10.5860/choice.184834 · Cited by 4,401 · see citing works
Formative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · see citing works
Inside the Black Box: Raising Standards through Classroom Assessment
Black, P., & Wiliam, D. (2010) · Phi Delta Kappan · doi:10.1177/003172171009200119 · Cited by 4,231 · see citing works
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,150 · see citing works
The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies
Dong, X., & McIntyre, S. H. (2014) · Quantitative Finance · doi:10.1080/14697688.2014.946440 · Cited by 1,681 · see citing works
Humphreys, L. G., & Lindquist, E. F. (1952) · The American Journal of Psychology · doi:10.2307/1418848 · Cited by 1,511 · see citing works
van der Linden, W. J., & van der Linden, W. J. (2010) · Springer eBooks · doi:10.1007/978-0-387-85461-8 · Cited by 515 · see citing works
Shermis, M. D., & Burstein, J. C. (2003) · Routledge eBooks · doi:10.4324/9781410606860 · Cited by 504 · see citing works
Handbook of Automated Essay Evaluation
(2013) · Routledge eBooks · doi:10.4324/9780203122761 · Cited by 491 · see citing works
(1966) · Phi Delta Kappan · Cited by 449 · see citing works
(2006) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 424 · see citing works
Linear Models for Optimal Test Design
van der Linden, W. J. (2005) · Springer eBooks · doi:10.1007/0-387-29054-0 · Cited by 295 · see citing works
Quellmalz, E., & Pellegrino, J. W. (2009) · Science · doi:10.1126/science.1168046 · Cited by 256 · see citing works
"Kappan Classic": Inside the Black Box--Raising Standards through Classroom Assessment.
Black, P., & Wiliam, D. (2010) · Phi Delta Kappan · Cited by 232 · see citing works
Automated Scoring and Annotation of Essays with the Intelligent Essay Assessor™
Shermis, M. D., & Burstein, J. C. (2003) · Routledge eBooks · doi:10.4324/9781410606860-15 · Cited by 153 · see citing works
(2012) · Routledge eBooks · doi:10.4324/9780203803912 · Cited by 152 · see citing works
(1995) · Phi Delta Kappan · Cited by 150 · see citing works
A survey on feature extraction for pattern recognition
Ding, S., Zhu, H. G., Jia, W., et al. (2011) · Artificial Intelligence Review · doi:10.1007/s10462-011-9225-y · Cited by 128 · see citing works
Automated Essay Scoring: Writing Assessment and Instruction
Shermis, M. D., Burstein, J. C., Higgins, D., & Zechner, K. (2010) · Elsevier eBooks · doi:10.1016/b978-0-08-044894-7.00233-5 · Cited by 115 · see citing works
How the Internet Will Help Large-Scale Assessment Reinvent Itself
Bennett, R. E. (2001) · Education Policy Analysis Archives · doi:10.14507/epaa.v9n5.2001 · Cited by 99 · see citing works
Applied Natural Language Processing: Identification, Investigation and Resolution
Hassel, M., & Dalianis, H. (2011) · Cited by 71 · see citing works
Contrasting State-of-the-Art Automated Scoring of Essays
Shermis, M. D., & Hamner, B. (2015) · Routledge eBooks · doi:10.4324/9780203122761.ch19 · Cited by 63 · see citing works
Applied Natural Language Processing
Hassel, M., & Dalianis, H. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-741-8 · Cited by 60 · see citing works
The E-rater® Automated Essay Scoring System
Burstein, J. C., Tetreault, J., & Madnani, N. (2015) · Routledge eBooks · doi:10.4324/9780203122761.ch4 · Cited by 57 · see citing works
LightSIDE: Open Source Machine Learning for Text
Burstein, J. C., Tetreault, J., Chodorow, M. S., et al. (2013) · Routledge eBooks · doi:10.4324/9780203122761-16 · Cited by 49 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242 · see citing works
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079 · see citing works
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087 · see citing works
Artificial Intelligence Revolutionizing the Field of Medical Education
Narayanan, S., Ramakrishnan, R., Durairaj, E., & Das, A. (2023) · Cureus · doi:10.7759/cureus.49604 · Cited by 164 · see citing works
Gustilo, L., Ong, E., & Lapinid, M. R. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00153-8 · Cited by 89 · see citing works
Artificial Intelligence developments in medical education: a conceptual and practical framework
Masters, K. (2020) · MedEdPublish · doi:10.15694/mep.2020.000239.1 · Cited by 25 · see citing works
Hollis-Sando, L., Pugh, C., Franke, K., Zerner, T., Tan, Y., Carneiro, G., Van den Hengel, A., Symonds, I., Duggan, P., & Bacchi, S. (2023) · Focus on Health Professional Education: A Multi-Professional Journal · doi:10.11157/fohpe.v24i1.531 · Cited by 7 · see citing works
- Authors
- Gierl, M. J., Latifi, S., Lai, H., Boulais, A., & De Champlain, A.
- Affiliations
- University of Alberta (CA)
- Medical Council of Canada (CA)
- Medical Council of Canada Ottawa Ontario Canada
- Corresponding author
- Gierl, Mark J
- Venue
- Medical Education
- Publisher
- Wiley
- Published
- 2014-09-09
- Volume
- 48
- Issue
- 10
- Pages
- 950-962
- ISSN
- 0308-0110 (ISSN-L)
- 1365-2923
- OpenAlex
- W2167507251
- Cited by
- 80
- References
- 31
- Citations by year
- 2026: 10 · 2025: 10 · 2024: 10 · 2023: 14 · 2022: 4 · 2021: 5 · 2020: 8 · 2019: 6 · 2018: 4 · 2017: 3 · 2016: 2 · 2015: 3 · 2014: 1
- FWCI
- 2.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Medical Education (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- automated essay scoring, medical education assessment, linguistic features, machine learning, rater agreement, formative assessment, summative assessment, high-stakes testing, student feedback, writing assessment
- MeSH terms
- Clinical Competence, Computer-Assisted Instruction, Education, Medical, Educational Measurement, Humans, Software, Writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
