Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Purpose: This study aimed to assess the performance of ChatGPT (GPT-3.5 and GPT-4) as a study tool in solving biostatistical problems and to identify any potential drawbacks that might arise from using ChatGPT in medical education, particularly in solving practical biostatistical problems.Methods: ChatGPT was tested to evaluate its ability to solve biostatistical problems from the Handbook of Medical Statistics by Peacock and Peacock in this descriptive study. Tables from the problems were transformed into textual questions. Ten biostatistical problems were randomly chosen and used as text-based input for conversation with ChatGPT (versions 3.5 and 4).Results: GPT-3.5 solved 5 practical problems in the first attempt, related to categorical data, cross-sectional study, measuring reliability, probability properties, and the t-test. GPT-3.5 failed to provide correct answers regarding analysis of variance, the chi-square test, and sample size within 3 attempts. GPT-4 also solved a task related to the confidence interval in the first attempt and solved all questions within 3 attempts, with precise guidance and monitoring.Conclusion: The assessment of both versions of ChatGPT performance in 10 biostatistical problems revealed that GPT-3.5 and 4’s performance was below average, with correct response rates of 5 and 6 out of 10 on the first attempt. GPT-4 succeeded in providing all correct answers within 3 attempts. These findings indicate that students must be aware that this tool, even when providing and calculating different statistical analyses, can be wrong, and they should be aware of ChatGPT’s limitations and be careful when incorporating this model into medical education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“To date, ChatGPT passed all 3 exams on the United States Medical Licensing Examination [4,5].”
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Exploring Students’ Perceptions of ChatGPT: Thematic Analysis and Follow-Up Survey
Shoufan, A. (2023) · IEEE Access · doi:10.1109/access.2023.3268224 · Cited by 499 · cited 2×
“These findings echo and confirm results from a recent publication emphasizing that students still must acquire subject foundations, maintain and develop critical thinking, and not unquestion-ingly trust chatbot-generated responses [11].”
Hassani, H., & Silva, E. S. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7020062 · Cited by 401
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.1 · Cited by 278
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195
Ibrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154
Generative AI (gAI) in medical education: Chat-GPT and co
Moritz, S., Romeike, B., Stosch, C., & Tolks, D. (2023) · German Medical Science GMS Publishing House · doi:10.3205/zma001636 · Cited by 37 · cited 1×
“It can substitute for a real discussion partner and provide unexpectedly good responses to requests and complex data [3].”
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T. H. (2023) · doi:10.1101/2023.05.13.23289943 · Cited by 10
Cited by
Xu, X., Chen, Y., & Miao, J. (2024) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2024.21.6 · Cited by 169
Lin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 58
ChatGPT for Univariate Statistics: Validation of AI-Assisted Data Analysis in Healthcare Research
Ruta, M. R., Gaidici, T., Irwin, C., & Lifshitz, J. (2025) · Journal of Medical Internet Research · doi:10.2196/63550 · Cited by 24
ChatGPT as a Tool for Biostatisticians: A Tutorial on Applications, Opportunities, and Limitations
Dobler, D., Binder, H., Boulesteix, A., Igelmann, J., Köhler, D., Mansmann, U., Pauly, M., Scherag, A., Schmid, M., Al Tawil, A., et al. (2025) · Statistics in Medicine · doi:10.1002/sim.70263 · Cited by 20
Shahrul, A. I., & Syed Mohamed, A. M. F. (2024) · Cureus · doi:10.7759/cureus.72581 · Cited by 9
Metadata
- Authors
- Ignjatović, A., & Stevanović, L.
- Venue
- Journal of Educational Evaluation for Health Professions
- Published
- 2023-10-16
- Volume
- 20
- Pages
- 28
- ISSN
- 1738-1339 (ISSN-L)
- 1975-5937
- OpenAlex
- W4387662141
- Cited by
- 38
- References
- 10
- FWCI
- 1.31
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study. https://app.cochat.ai/papers/10.3352/jeehp.2023.20.28
- The paper
- Ignjatović, A., & Stevanović, L. (2023). Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2023.20.28
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.