CoChat
Sign Up
Cited by 42Educational Games and GamificationInnovations in Medical EducationArtificial Intelligence in Healthcare and Education

Chatbot-based serious games: a useful tool for training medical students? A randomized controlled trial

Open access
Al Kahf, Salma, Roux, Baptiste, Clerc, Sebastien, Bassehila, Mona, Lecomte, A., Moncomble, Elsa, Alabadan, Elodie, de Montmolin, Nina +8 more·2023·PLoS ONE
DOI 10.1371/journal.pone.0278673PMC PMC10010502
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVES: Chatbots, conversational agents that walk medical students (MS) though a clinical case, are serious games that seem to be appreciated by MS. Their impact on MS's performance in exams however was not yet evaluated. Chatprogress is a chatbot-based game developed at Paris Descartes University. It contains 8 pulmonology cases with step-by-step answers delivered with pedagogical comments. The CHATPROGRESS study aimed to evaluate the impact of Chatprogress on students' success rate in their end-term exams. METHODS: We conducted a post-test randomized controlled trial held on all fourth-year MS at Paris Descartes University. All MS were asked to follow the University's regular lectures, and half of them were randomly given access to Chatprogress. At the end of the term, medical students were evaluated on pulmonology, cardiology and critical care medicine. MAIN OUTCOMES MEASURES: The primary aim was to evaluate an increase in scores in the pulmonology sub-test for students who had access to Chatprogress, compared to those who didn't. Secondary aims were to evaluate an increase in scores in the overall test (Pulmonology, Cardiology and Critical care medicine test (PCC)) and to evaluate the correlation between access to Chatprogress and overall test score. Finally, students' satisfaction was assessed using a survey. RESULTS: From 10/2018 to 06/2019, 171 students had access to Chatprogress (the Gamers) and among them, 104 ended up using it (the Users). Gamers and Users were compared to 255 Controls with no access to Chatprogress. Differences in scores on the pulmonology sub-test over the academic year were significantly higher among Gamers and Users vs Controls (mean score: 12.7/20 vs 12.0/20, p = 0.0104 and mean score: 12.7/20 vs 12.0/20, p = 0.0365 respectively). This significant difference was present as well in the overall PCC test scores: (mean score: 12.5/20 vs 12.1/20, p = 0.0285 and 12.6/20 vs 12.1/20, p = 0.0355 respectively). Although no significant correlation was found between the pulmonology sub-test's scores and MS's assiduity parameters (number of finished games among the 8 proposed to Users and number of times a User finished a game), there was a trend to a better correlation when users were evaluated on a subject covered by Chatprogress. MS were also found to be fans of this teaching tool, asking for more pedagogical comments even when they got the questions right. CONCLUSION: This randomised controlled trial is the first to demonstrate a significant improvement in students' results (in both the pulmonology subtest and the overall PCC exam) when they had access to Chatbots, and even more so when they actually used it.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0278673

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The primary benefits of using chatbots include integrating content, quick access, allowing multiple users, and immediate assistance.

    “The primary benefits of using chatbots include integrating content, quick access, allowing multiple users and immediate assistance [9,10].”

    From Artificial Intelligence Revolutionizing the Field of Medical Education (2023) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [6]Okonkwo C.W. & Ade-Ibijola A. (2021). Chatbots applications in education: A systematic review, Computers and Education: Artificial Intelligence, 2, 100033.
  2. [10]Smutny Pavel, Schreiberova Petra, Chatbots for learning: A review of educational chatbots for the Facebook Messenger, Computers & Education, Volume 151, 2020, 103862, ISSN 0360-1315.

Cited by

Sorted by citations

Related

Authors
Al Kahf, S., Roux, B., Clerc, S., Bassehila, M., Lecomte, A., Moncomble, E., Alabadan, E., de Montmolin, N., Jablon, E., François, E., Friedlander, G., Badoual, C., Meyer, G., Roche, N., Martin, C., & Planquette, B.
Affiliations
  • Université Paris Cité (FR)
  • Assistance Publique – Hôpitaux de Paris (FR)
Corresponding author
Planquette, Benjamin
ORCID
Venue
PLoS ONE
Publisher
Public Library of Science (PLoS)
Published
2023-03-13
Volume
18
Issue
3
Pages
e0278673
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0278673
PMC
PMC10010502
OpenAlex
W4324019775
Cited by
42
References
17
Citations by year
2026: 9 · 2025: 13 · 2024: 13 · 2023: 7
FWCI
20.34
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Accompagnement à la Gestion de l’Innovation pour la Réussite des étudiants
Topics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
chatbots, serious games, medical education, respiratory medicine, intensive care, student satisfaction, gamification, ChatGPT
MeSH terms
Education, Medical, Undergraduate, Educational Measurement, Humans, Software, Students, Medical, Video Games

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review