Usability and Efficacy of Artificial Intelligence Chatbots (ChatGPT) for Health Sciences Students: Protocol for a Crossover Randomized Controlled Trial
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Background: The integration of artificial intelligence (AI) into health sciences students’ education holds significant importance. The rapid advancement of AI has opened new horizons in scientific writing and has the potential to reshape human-technology interactions. AI in education may impact critical thinking, leading to unintended consequences that need to be addressed. Understanding the implications of AI adoption in education is essential for ensuring its responsible and effective use, empowering health sciences students to navigate AI-driven technologies’ evolving field with essential knowledge and skills. Objective: This study aims to provide details on the study protocol and the methods used to investigate the usability and efficacy of ChatGPT, a large language model. The primary focus is on assessing its role as a supplementary learning tool for improving learning processes and outcomes among undergraduate health sciences students, with a specific emphasis on chronic diseases. Methods: This single-blinded, crossover, randomized, controlled trial is part of a broader mixed methods study, and the primary emphasis of this paper is on the quantitative component of the overall research. A total of 50 students will be recruited for this study. The alternative hypothesis posits that there will be a significant difference in learning outcomes and technology usability between students using ChatGPT (group A) and those using standard web-based tools (group B) to access resources and complete assignments. Participants will be allocated to sequence AB or BA in a 1:1 ratio using computer-generated randomization. Both arms include students’ participation in a writing assignment intervention, with a washout period of 21 days between interventions. The primary outcome is the measure of the technology usability and effectiveness of ChatGPT, whereas the secondary outcome is the measure of students’ perceptions and experiences with ChatGPT as a learning tool. Outcome data will be collected up to 24 hours after the interventions. Results: This study aims to understand the potential benefits and challenges of incorporating AI as an educational tool, particularly in the context of student learning. The findings are expected to identify critical areas that need attention and help educators develop a deeper understanding of AI’s impact on the educational field. By exploring the differences in the usability and efficacy between ChatGPT and conventional web-based tools, this study seeks to inform educators and students on the responsible integration of AI into academic settings, with a specific focus on health sciences education. Conclusions: By exploring the usability and efficacy of ChatGPT compared with conventional web-based tools, this study seeks to inform educators and students about the responsible integration of AI into academic settings. Trial Registration: ClinicalTrails.gov NCT05963802; https://clinicaltrials.gov/study/NCT05963802 International Registered Report Identifier (IRRID): PRR1-10.2196/51873
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 4×
“RenderX and academia, to maximize their potential while addressing limitations and risks [8].”
BackgroundInfluentialHaenlein, M., & Kaplan, A. (2019) · California Management Review · doi:10.1177/0008125619864925 · Cited by 2,655
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842
Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips
Sparrow, B., Liu, J., & Wegner, D. M. (2011) · Science · doi:10.1126/science.1207745 · Cited by 1,762
Ethical principles for artificial intelligence in education
Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B.-P. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11316-w · Cited by 1,228 · cited 1×
“The integration of artificial intelligence (AI) into the education field has led to many possibilities, benefiting both educators and students [1].”
BackgroundChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
CONSORT 2010 statement: extension to randomised crossover trials
Dwan, K., Li, T., Altman, D. G., & Elbourne, D. (2019) · BMJ · doi:10.1136/bmj.l4378 · Cited by 785
Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020) · BMJ · doi:10.1136/bmj.m3164 · Cited by 702
The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study
Zhou, L., Bao, J., Setiawan, I. M. A., Saptono, A., & Parmanto, B. (2019) · JMIR mHealth and uHealth · doi:10.2196/11500 · Cited by 675 · cited 1×
“The SUS is commonly used in numerous studies to assess the effectiveness and usability of new technologies, including in the education field [27-29].”
MethodUnlocking the opportunities through ChatGPT Tool towards ameliorating the education system
Javaid, M., Haleem, A., Singh, R. P., Khan, S., & Khan, I. H. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100115 · Cited by 535 · cited 2×
“Educators can integrate ChatGPT into their courses to customize the learning journey for their students by considering their preferred learning methods and existing proficiency levels [4].”
BackgroundYu, H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1181712 · Cited by 409 · cited 1×
“Concerns have been raised regarding its potential to promote academic dishonesty, stifle creativity, and diminish critical thinking abilities [7].”
BackgroundArtificial intelligence in medical education: a cross-sectional needs assessment
Civaner, M. M., Uncu, Y., Bulut, F., Chalil, E. G., & Tatli, A. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03852-3 · Cited by 363
Alshater, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4312358 · Cited by 304
An exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · PLOS ONE · doi:10.1371/journal.pone.0292216 · Cited by 238
An exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · doi:10.1101/2023.03.31.23287979 · Cited by 238
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234 · cited 1×
“They emphasized the need for more formal experiments to rigorously gauge the user experience, leveraging these findings to design guidelines that prioritize usability [12].”
BackgroundMoldt, J.-A., Festl-Wietek, T., Madany Mamlouk, A., Nieselt, K., Fuhl, W., & Herrmann-Werner, A. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2182659 · Cited by 186 · cited 3×
“Students expressed a desire for more comprehensive AI education integrated into their medical curriculum, emphasizing the importance of preparing them for AI’s role in health care [35].”
BackgroundAssessing the Usability of ChatGPT for Formal English Language Learning
Shaikh, S., Yayilgan, S. Y., Klimova, B., & Pikhart, M. (2023) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe13090140 · Cited by 183
Ethical Management of Artificial Intelligence
Brendel, A. B., Mirbabaie, M., Lembcke, T.-B., & Hofeditz, L. (2021) · Sustainability · doi:10.3390/su13041974 · Cited by 174
Mir, M. M. (2023) · PubMed · doi:10.30476/jamp.2023.98655.1803 · Cited by 165 · cited 1×
“AI takes on various roles as an intelligent tutor, a collaborative learning partner, a valuable tool, and even an adviser in influencing educational policies [2,3].”
BackgroundPucchio, A., Rathagirishnan, R., Caton, N., Gariscsak, P. J., Del Papa, J., Nabhen, J. J., Vo, V., Lee, W., & Moraes, F. Y. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03896-5 · Cited by 138 · cited 4×
“In recent years, there has been a growing body of research on AI, focusing on its applications in various fields, including medical education, health care, radiography, and ophthalmology [5,33,34].”
BackgroundInfluentialEstablishing the concept of AI literacy: Focusing on competence and purpose
Yi, Y. (2021) · JAHR · doi:10.21860/j.12.2.8 · Cited by 113
Cited by
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
Li, J., Zong, H., Wu, E., Wu, R., Peng, Z., Zhao, J., Yang, L., Xie, H., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05738-y · Cited by 104
“Moreover, it can automatically evaluate and score the level of medical writing, and provide modification suggestions for improvement [18].”
BackgroundApplication of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
Elhassan, S. E., Sajid, M. R., Syed, A. M., Fathima, S. A., Khan, B. S., & Tamim, H. (2025) · JMIR Medical Education · doi:10.2196/63065 · Cited by 39
Assessment of Large Language Models (LLMs) in Decision-Making Support for Gynecologic Oncology
Gumilar, K. E., Indraprasta, B. R., Faridzi, A. S., Wibowo, B. M., Herlambang, A., Rahestyningtyas, E., Irawan, B., Tambunan, Z., Bustomi, A. F., Brahmantara, B. N., et al. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.10.050 · Cited by 31
Metadata
- Authors
- Veras, M., Dyer, J.-O., Rooney, M., Barros Silva, P. G., Rutherford, D., & Kairy, D.
- Venue
- JMIR Research Protocols
- Publisher
- JMIR Publications Inc.
- Published
- 2023-11-24
- Volume
- 12
- Pages
- e51873
- ISSN
- 1929-0748 (ISSN-L)
- OpenAlex
- W4387828398
- Cited by
- 47
- References
- 23
- FWCI
- 1.63
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.