CoChat
Sign Up
Cited by 36Artificial Intelligence in Healthcare and EducationInnovations in Medical EducationArtificial Intelligence in Education

AI in medical education: uses of AI in construction type A MCQs

Open access
Rezigalla, Assad Ali·2024·BMC Medical Education
DOI 10.1186/s12909-024-05250-3PMC PMC10919037
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The introduction of competency-based education models, student centers, and the increased use of formative assessments have led to demands for high-quality test items to be used in assessments. This study aimed to assess the use of an AI tool to generate MCQs type A and evaluate its quality. Methods: The study design was cross-sectional analytics conducted from June 2023 to August 2023. This study utilized formative TBL. The AI tool (ChatPdf.com) was selected to generate MCQs type A. The generated items were evaluated using a questionnaire for subject experts and an item (psychometric) analysis. The questionnaire to the subject experts about items was formed based on item quality and rating of item difficulty. Results: The total number of recurrent staff members as experts was 25, and the questionnaire response rate was 68%. The quality of the items ranged from good to excellent. None of the items had scenarios or vignettes and were direct. According to the expert’s rating, easy items represented 80%, and only two had moderate difficulty (20%). Only one item out of the two moderate difficulties had the same difficulty index. The total number of students participating in TBL was 48. The mean mark was 4.8 ± 1.7 out of 10. The KR20 is 0.68. Most items were of moderately difficult (90%) and only one was difficult (10%). The discrimination index of the items ranged from 0.77 to 0.15. Items with excellent discrimination represented 50% (5), items with good discrimination were 3 (30%), and only one time was poor (10%), and one was none discriminating. The non-functional distractors were 26 (86.7%), and the number of non-functional distractors was four (13.3%). According to distractor analysis, 60% of the items were excellent, and 40% were good. A significant correlation ( p = 0.4, r = 0.30) was found between the difficulty and discrimination indices. Conclusion: Items constructed using AI had good psychometric properties and quality, measuring higher-order domains. AI allows the construction of many items within a short time. We hope this paper brings the use of AI in item generation and the associated challenges into a multi-layered discussion that will eventually lead to improvements in item generation and assessment in general.

AI snapshot

Written from the paper’s full text.

TL;DR

This cross-sectional study assessed type A MCQs generated by ChatPdf.com from an anatomy PDF and used in a formative TBL i-Rat. The 10 AI-generated items showed acceptable psychometric properties and good-to-excellent expert-rated quality, though the sample and item set were small.

Key findings

  • Expert review (25 experts, 68% response rate) rated item quality as good to excellent; none had scenarios or vignettes and all were direct; 80% were rated easy and 20% moderately difficult.
  • Among 48 students, the mean i-Rat mark was 4.8 ± 1.7 out of 10; KR20 was 0.68; the average difficulty index was 47.7 ± 4.8 (moderate), with 90% of items moderately difficult and 10% difficult.
  • The discrimination index ranged from 0.77 to 0.15, with a mean of 0.41 ± 0.08 (excellent); 50% (5) of items had excellent discrimination, 30% (3) good, 10% poor, and 10% non-discriminating.
  • Distractor analysis of 30 distractors reported 26 (86.7%) non-functional distractors and 4 (13.3%) NFDs; 60% of items were excellent and 40% good. A significant correlation was reported between difficulty and discrimination indices (p = 0.4, r = 0.30).

Methods

  • Cross-sectional analytic study conducted June–August 2023 in the Department of Basic Medical Sciences (anatomy); sampling was total coverage of students registered to the musculoskeletal course (2022–2023), with 48 participants.
  • ChatPdf.com was selected using criteria including availability, free cost, ease of use, PDF format, safety, and confidentiality; a PDF section on the posterior abdominal wall from a standard anatomy book was uploaded, and the tool was ordered to construct type A MCQs.
  • The 10 generated items were used in the i-Rat of a formative TBL; evaluation used an expert questionnaire adapted from prior work and modified Angoff difficulty ratings, plus psychometric item analysis in SPSS v25 including KR20, difficulty index, discrimination index, distractor efficiency, and Pearson correlation.

Limitations

  • The paper states: small sample size (48 participants), small number of items, TBL study context, and no test of whether the AI tool would generate the same questions on other occasions.
  • Evident from the text: the study was limited to a single course, topic, and institution, with no comparison against traditionally constructed items; the expert questionnaire response rate was 68%.
  • The text also shows inconsistent naming of the AI tool (ChatPdf.com vs ChatPDF.ai) and contradictory reporting of non-functional distractor counts, which limits clarity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12909-024-05250-3

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 54 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The quality of the generated items ranged from good to excellent according to the expert questionnaire.

    “The quality of the items ranged from good to excellent.”
    • Also stated in: § The questionnaire to experts · plain
      “According to the questionnaire responses, the item quality was good to excellent.”
    Quote in the paperSubstantial claim
  • Findingown data

    None of the generated items had scenarios or vignettes; they were direct.

    “None of the items had scenarios or vignettes and were direct.”
    • Also stated in: § The questionnaire to experts · plain
      “None of the items had scenarios or vignettes and were direct.”
    Quote in the paperNot a substantial claim
  • Findingown data

    A significant correlation was found between the difficulty and discrimination indices (p = 0.4, r = 0.30).

    “A significant correlation (p = 0.4, r = 0.30) was found between the difficulty and discrimination indices.”
    • Also stated in: § Item analysis · plain
      “The Pearson correlation test revealed a significant correlation between the difficulty and discrimination indices (p = 0.4, r = 0.30).”
    Quote in the paperNot a substantial claim
  • Findingown data

    The mean mark of students was 4.8 ± 1.7 out of 10.

    “The mean mark was 4.8 ± 1.7 out of 10.”
    • Also stated in: § Item analysis · plain
      “The mean mark was 4.8 ± 1.7 out of 10.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Only one of the two moderate difficulty items had the same difficulty index as the expert rating.

    “Only one item out of the two moderate difficulties had the same difficulty index.”
    • Also stated in: § The questionnaire to experts · plain
      “Only one item of the two moderate difficulties had the same difficulty index, while the rest had expert ratings that were different from the item difficulty index.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 45

  • May combine two claims.

    Findingown data

    Most items were moderately difficult (90%) and only one was difficult (10%).

    “Most items were of moderately difficult (90%) and only one was difficult (10%).”
    • Also stated in: § Item analysis · plain
      “Most items were moderately difficult (90%) and only one difficult (10%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    According to distractor analysis, 60% of the items were excellent and 40% were good.

    “According to distractor analysis, 60% of the items were excellent, and 40% were good.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    According to expert ratings, easy items represented 80% and only two had moderate difficulty (20%).

    “According to the expert’s rating, easy items represented 80%, and only two had moderate difficulty (20%).”
    • Also stated in: § Results · plain
      “Easy items represented 80%, and only two had moderate difficulty (20%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Items with excellent discrimination represented 50% (5), good discrimination 30% (3), poor 10% (1), and none discriminating 1.

    “Items with excellent discrimination represented 50% (5), items with good discrimination were 3 (30%), and only one time was poor (10%), and one was none discriminating.”
    • Also stated in: § Item analysis · plain
      “Items with excellent discrimination represented 50% (5), items with good discrimination were 3 (30%), and only one time was poor (10%), and one was none discriminating.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All items tested knowledge and passed the hand cover test.

    “All items tested knowledge and passed the hand cover test.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The distractor and correct options followed grammatical and logical forms.

    “The distractor and correct options followed the grammatical and logical forms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The generated items are devoid of absolute words and "none of the above".

    “The generated items are devoid of absolute words and “none of the above’ (Fig. 2).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The non-discriminating item had a moderately difficult index.

    “The non-discriminating item had a moderately difficult index.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Only one item contained a repetition of words in the stem "clang, clue," and its difficulty and discrimination indexes were within acceptable levels.

    “Only one item contained a repetition of words in the stem “clang, clue,” and its difficulty and discrimination indexes were within acceptable levels.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    More than half of the items had excellent discrimination and function distractors.

    “more than half had excellent discrimination and function distractors”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The average rating of items by experts was easy.

    “The average rating of items by experts was easy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The generated items have acceptable average levels of difficulty index and discrimination index.

    “The generated items have acceptable average levels of difficulty index and discrimination index.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The students' scores tended to cluster around the mean, with low dispersion between the minimum and maximum scores.

    “The students’ scores tended to cluster around the mean, with low dispersion between the minimum and maximum scores.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The difficulty index of the items was moderate, and only one item was difficult.

    “The difficulty index of the items was moderate, and only one item was difficult.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    In the current study, experts rated the items as easy, and only two were moderately difficult.

    “In the current study, experts rated the items as easy, and only two were moderately difficult.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Currently, the items generated by AI have acceptable levels of difficulty index, discrimination, and distractor analysis.

    “Currently, the items generated by AI have acceptable levels of difficulty index, discrimination, and distractor analysis.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The total number of students participating in TBL was 48.

    “The total number of students participating in TBL was 48.”
    • Also stated in: § Results · plain
      “The total number of student participants was 48.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Items constructed using AI had good psychometric properties and quality, measuring higher-order domains.

    “Items constructed using AI had good psychometric properties and quality, measuring higher-order domains.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The rating of item difficulty was assessed based on modified Angoff methods.

    “The rating of item difficulty was assessed based on modified Angoff methods (Table 1) [33, 34].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [33] Rezigalla AA. Angoff’s method: the impact of raters’ selection. Saudi J Med Med Sci. 2015;3(3):220–5. source

      Not checked in the free run.

    • cites [34] Joseph MN, Chang J, Buck SG, Auerbach MA, Wong AH, Beardsley TD, Reeves PM, Ray JM, Evans LV. A novel application of the modified Angoff method to rate case difficulty in simulation-based research. Simul Healthc. 2021;16(6):e142–50. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The questionnaire response rate was 68%.

    “the questionnaire response rate was 68%.”
    • Also stated in: § The questionnaire to experts · plain
      “the questionnaire response rate was 68%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The TBL and topic were chosen to ensure one-dimensionality of the content material, which can affect internal consistency.

    “The TBL and topic were chosen to ensure the one-dimensionality of the content material, which can affect internal consistency.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The growing need for test items requires new advancements in item construction and generation in addition to the traditional method.

    “The growing need for test items requires new advancements in item construction and generation in addition to the traditional method [1].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1] Pugh D, De Champlain A, Gierl M, Lai H, Touchie C. Can automated item generation be used to develop high quality MCQs that assess application of knowledge? Res Pract Technol Enhanced Learn. 2020;15:1–13. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Some existing AI tools for item generation are costly, need high technical support, and have good results.

    “Some of these are costly, need high technical support, and have good results [24–27].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [24] Wu F, Lu C, Zhu M, Chen H, Zhu J, Yu K, Li L, Li M, Chen Q, Li X. Towards a new generation of artificial intelligence in China. Nat Mach Intell. 2020;2(6):312–6. source

      Not checked in the free run.

    • cites [25] Sounderajah V, Ashrafian H, Golub RM, Shetty S, De Fauw J, Hooft L, Moons K, Collins G, Moher D, Bossuyt PM. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ open. 2021;11(6):e047709. source

      Not checked in the free run.

    • cites [26] Choi J. Automatic item generation with machine learning techniques. In: Application of Artificial Intelligence to Assessment edn. Edited by Jiao H, Cissitz R. USA: Information Age Publishing; 2020: 189–210.

      Not checked in the free run.

    • cites [27] Kurdi G, Leo J, Parsia B, Sattler U, Al-Emari S. A systematic review of automatic question generation for educational purposes. Int J Artif Intell Educ. 2020;30:121–204. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Inter-rater agreement for expert rating was limited to 30%, and deviating experts were asked to rerate.

    “Inter-rater agreement for the expert rating was limited to 30%. In cases of deviation in ratings, the concerned expert is requested to rerate according to other expert ratings or keep his rates [33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Rezigalla AA. Angoff’s method: the impact of raters’ selection. Saudi J Med Med Sci. 2015;3(3):220–5. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The total number of recurrent staff members as experts was 25.

    “The total number of recurrent staff members was 25.”
    • Also stated in: § The questionnaire to experts · plain
      “The total number of participating staff members as experts was 25”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisereasoning

    Through AI, it will be easier and more feasible to construct and update items and form item banks.

    “Through AI, it will be easier and more feasible to construct and update items (questions) and form item banks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The questionnaire to subject experts was based on and adapted from the work of Susan and David (2016) and other authors.

    “The questionnaire to the subject experts about item construction and quality was based on and adapted from the work of Susan and David (2016) and other authors [2, 4, 32] (Table 1).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [2] Naidoo M. The pearls and pitfalls of setting high-quality multiple choice questions for clinical medicine. South Afr Family Practice: Official J South Afr Acad Family Practice/Primary Care. 2023;65(1):e1–e4.

      Not checked in the free run.

    • cites [4] Rahim MF, Bham SQ, Khan S, Ansari T, Ahmed M. Improving the quality of MCQs by enhancing cognitive level and using psychometric analysis: improving the quality of MCQs by enhancing cognitive level. Pakistan J Health Sci 2023:115–21. source

      Not checked in the free run.

    • cites [32] Case SM, Swanson DB. Writing one-best-answer questions for the Basic and Clinical sciences. Constructing written test questions for the Basic and Clinical sciences. 3 ed. Philadelphia: National Board of Medical Examiners; 2016. pp. 31–66.

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The target group of experts was selected according to criteria: staff members in medical colleges within KSA, experience in teaching human anatomy, experience in medical education, and work in a medical college with similar/equivalent curriculum.

    “The target group of experts was selected according to the following criteria: staff members in medical colleges (within the KSA), experience in teaching human anatomy, experience in medical education, and work in a medical college with a similar or equivalent curriculum (problem-based, SPICE).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The questionnaire was tested in a pilot study, and pilot study experts/results were not included in the analysis.

    “The questionnaire was tested in a pilot study. The experts included in the pilot study, or the results of the pilot study were not included in the analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Items with deviated parameters in item analysis were checked for possible content, technical, or grammatical flaws.

    “Items with deviated parameters in item analysis (one difficult item, one non-discriminating item, and one with poor discrimination power) were checked for possible content, technical, or grammatical flaws.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study was approved by the Research and Ethics Committee of the College of Medicine, University of Bisha (KSA).

    “The study was approved by the Research and Ethics Committee of the College of Medicine, University of Bisha (KSA).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The difficulty index ranges from 0 to 100, where higher values indicate easier questions.

    “The difficulty index ranges from 0 to 100, where higher values indicate easier questions and lower values represent the difficulty of hard items.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] Rezigalla AA. Item analysis: Concept and application. In: Medical Education for the 21st Century edn. Edited by Firstenberg MS, Stawicki SP: IntechOpen; 2022: 105–120.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Item analysis provides feedback about item construction and its validity and reliability after it appears in an examination.

    “Item analysis provides feedback about item construction and its validity and reliability after it appears in an examination [5, 35, 36].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [5] Licona-Chávez AL, Velázquez-Liaño LR. Quality assessment of a multiple choice test through psychometric properties. MedEdPublish. 2020;9(91):1–12.

      Not checked in the free run.

    • cites [35] Kumar D, Jaipurkar R, Shekhar A, Sikri G, Srinivas V. Item analysis of multiple choice questions: a quality assurance test for an assessment tool. Med J Armed Forces India. 2021;77:85–S89. source

      Not checked in the free run.

    • cites [36] Adiga MNS, Acharya S, Holla R. Item analysis of multiple-choice questions in pharmacology in an Indian Medical School. J Health Allied Sci NU. 2021;11(03):130–5. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The DIS ranged from 1.0 to − 1.0.

    “The DIS ranged from 1.0 to − 1.0.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [35] Kumar D, Jaipurkar R, Shekhar A, Sikri G, Srinivas V. Item analysis of multiple choice questions: a quality assurance test for an assessment tool. Med J Armed Forces India. 2021;77:85–S89. source

      Not checked in the free run.

    • cites [44] Date AP, Borkar AS, Badwaik RT, Siddiqui RA, Shende TR, Dashputra AV. Item analysis as tool to validate multiple choice question bank in pharmacology. Int J Basic Clin Pharmacol. 2019;8(9):1999–2003. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The range of item difficulty can be categorized as difficult (< 39), moderate (40-80), and easy (> 80).

    “The range of item difficulty can be categorized as difficult (< 39), moderate (40–80), and easy (> 80) [37, 43].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [37] Rezigalla AA. Item analysis: Concept and application. In: Medical Education for the 21st Century edn. Edited by Firstenberg MS, Stawicki SP: IntechOpen; 2022: 105–120.

      Not checked in the free run.

    • cites [43] Kaur M, Singla S, Mahajan R. Item analysis of in use multiple choice questions in pharmacology. Int J Appl Basic Med Res. 2016;6(3):70–173. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Students were informed that TBL was formative and participation did not impact grades or GPA.

    “The students were informed that the TBL was formative and that their participation in the study did not impact their grades or grade point average (GPA)GPA.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Values > 0.7, 0.6-0.7 is acceptable, 0.5-0.6 poor, < 0.5, and < 0.30 unreliable.

    “It was reported that values > 0.7, 0.6–0.7 is acceptable, 0.5–0.6. poor, < 0.5, and < 0.30, unreliable [42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] Hassan S, Hod R. Use of item analysis to improve the quality of single best answer multiple choice question in summative assessment of undergraduate medical students in Malaysia. Educ Med J. 2017;9(3):33–43. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Written consent was obtained from participating students.

    “written consent was obtained from the participating students as part of the i-Rat.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A value of 0.7 was reported as acceptable for a short test (less than 50 items) and KR20 of 0.8 for a large exam (more than 50 items).

    “A value of 0.7 was reported as acceptable for a short test (less than 50 items) and KR20 of 0.8 for a large exam (more than 50 items test) [41].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41] Bell BA. Pretest–Posttest Design. In: Encyclopedia of research design. Volume 2, edn. Edited by Salkind NJ. Thousand Oaks: SAGE Publications, Inc.; 2014: 1087–1092.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The TBL content was conducted according to the standards of TBL conduction [29].

    “The TBL content was a single anatomical topic and was conducted according to the standards of TBL conduction [29].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] El-Ashkar A, Aboregela A, Alam-Eldin Y, Metwally A. Team-based learning as an inspiring tool for teaching parasitology in the integrated curricula. Parasitologists United J. 2023;16(1):64–72. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The test time and number of items in the current study were in accordance with recommendations.

    “The test time and number of items in the current study were in accordance with recommendations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The KR20 is reported to be affected by many factors such as exam time, the number and inter-relation of the items (dimensionality), item difficulty, variations in examinee responses, and the number of examinees [37].

    “The KR20 is reported to be affected by many factors such as exam time, the number and inter-relation of the items (dimensionality), item difficulty, variations in examinee responses, and the number of examinees [37].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [37] Rezigalla AA. Item analysis: Concept and application. In: Medical Education for the 21st Century edn. Edited by Firstenberg MS, Stawicki SP: IntechOpen; 2022: 105–120.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Such findings were reported in anatomy exam [48], community Medicine [49], and multidisciplinary summative exams [50, 51] that were constructed traditionally.

    “Such findings were reported in anatomy exam [48], community Medicine [49], and multidisciplinary summative exams [50, 51] that were constructed traditionally.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [48] D’Sa JL, Visbal-Dionaldo ML. Analysis of multiple choice questions: Item Difficulty, discrimination index and distractor efficiency. Int J Nurs Educ. 2017;9(3):109–14. source

      Not checked in the free run.

    • cites [49] Gajjar S, Sharma R, Kumar P, Rana M. Item and test analysis to identify quality multiple choice questions (MCQs) from an assessment of medical students of Ahmedabad, Gujarat. Indian J Community Medicine: Official Publication Indian Association Prev Social Med. 2014;39(1):17–20. source

      Not checked in the free run.

    • cites [50] Mitra N, Nagaraja H, Ponnudurai G, Judson J. The levels of difficulty and discrimination indices in type a multiple choice questions of pre-clinical semester 1 multidisciplinary summative tests. IeJSME. 2009;3(1):2–7. source

      Not checked in the free run.

    • cites [51] Kumar D, Jaipurkar R, Shekhar A, Sikri G, Srinivas V. Item analysis of multiple choice questions: a quality assurance test for an assessment tool. Med J Armed Forces India. 2021;1(77):85–S89. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Students could attend TBL without including their responses in the study.

    “In addition, the students could attend TBL without including their responses in the study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Permission to use the AI tool was obtained through the support team.

    “Permission to use the AI tool was obtained through the support team for the tool.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The range of DIS can be categorized into excellent (≥ 0.35), Good (0.25-0.34), Acceptable (0.21-0.24), and poor (≤ 0.20).

    “The range of DIS can be categorized into excellent (≥ 0.35), Good (0.25–0.34), Acceptable (0.21–0.24), and poor (≤ 0.20) [45, 46].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [45] Elfaki OA, Bahamdan KA, Al-Humayed S. Evaluating the quality of multiple-choice questions used for final exams at the Department of Internal Medicine, College of Medicine, King Khalid University. Sudan Med Monit. 2015;10(4):123–7. source

      Not checked in the free run.

    • cites [46] Rezigalla AA. Item analysis: Concept and application. In: Medical Education for the 21st Century edn. Edited by Firstenberg MS, Stawicki SP. London: Intechopen; 2022: 1–16.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The clustering of students' scores may indicate that students' performance and competence are equal or that the items evaluate students' knowledge well.

    “Such clustering of students’ scores may indicate that students’ performance and competence are equal or that the items evaluate students’ knowledge well.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The i-Rat scores reflect students' performance and competence.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A positive DIS indicates that high achievers answered the item more correctly than lower achievers.

    “A positive DIS indicates that high achievers answered the item more correctly than lower achievers and vice versa [35, 44].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [35] Kumar D, Jaipurkar R, Shekhar A, Sikri G, Srinivas V. Item analysis of multiple choice questions: a quality assurance test for an assessment tool. Med J Armed Forces India. 2021;77:85–S89. source

      Not checked in the free run.

    • cites [44] Date AP, Borkar AS, Badwaik RT, Siddiqui RA, Shende TR, Dashputra AV. Item analysis as tool to validate multiple choice question bank in pharmacology. Int J Basic Clin Pharmacol. 2019;8(9):1999–2003. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A negative correlation between difficulty and discrimination indices indicates an association between easy items and a high discrimination index.

    “A negative correlation between difficulty and discrimination indices indicates an association between easy items and a high discrimination index.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The total number of participants was 48.

    “The total number of participants was 48.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Items with deviated parameters of item analysis were checked, and none contained any content, technical, or grammatical flaws.

    “Items with deviated parameters of item analysis were checked, and none contained any content, technical, or grammatical flaws.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Logically, difficult items can discriminate between students and have a high discrimination index.

    “Logically, difficult items can discriminate between students and have a high discrimination index.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Difficult items with no flaws are answered by high-achieving rather than low-achieving students.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Easy items answered by most examinees will not discriminate between them.

    “Meanwhile, the easy items answered by most examinees will not discriminate between them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Alareifi (2023) described a strong negative correlation between difficulty and discrimination indices (p = 0.00, r= -0.936) [57].

    “Alareifi (2023) described a strong negative correlation between difficulty and discrimination indices (p = 0.00, r= -0.936) [57].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57] Alareifi RM. Analysis of MCQs in summative exam in English: Difficulty Index, discrimination index and relationship between them. J Eduction Hum Sci. 2023;20:124–35.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other authors have reported a significant linear and dome-shaped relationship between difficulty and the discrimination index [6, 56].

    “Other authors have reported a significant linear and dome-shaped relationship between difficulty and the discrimination index [6, 56].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] Ramzan M, Khan KW, Bibi S, Imran SS. Difficulty and discrimination analysis of end of term multiple-choice questions at Community Medicine Department, Wah Medical College. Pakistan Armed Forces Med J. 2021;71(4):1308–10. source

      Not checked in the free run.

    • cites [56] Sim S-M, Rasiah RI. Relationship between item difficulty and discrimination indices in true/false-type multiple choice questions of a para-clinical multidisciplinary paper. Annals-Academy Med Singap. 2006;35(2):67–81. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The recommended percentage of items for an ideal difficulty-balanced exam is 5% for easy items, 5% for difficult items, 20% for moderately easy, 20% for moderately difficult, and 50% for average ones [5].

    “The recommended percentage of items for an ideal difficulty-balanced exam is 5% for easy items, 5% for difficult items, 20% for moderately easy, 20% for moderately difficult, and 50% for average ones [5].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Licona-Chávez AL, Velázquez-Liaño LR. Quality assessment of a multiple choice test through psychometric properties. MedEdPublish. 2020;9(91):1–12.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Despite one item that is not discriminating, the test has excellent quality to discriminate between higher- and lower-achieving students.

    “Despite one item that is not discriminating, the test has excellent quality to discriminate between higher- and lower-achieving students.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The test had 10 items.

    “Despite the few items (10)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    These reports indicate that an increase in DI is associated with an increase in the discrimination index.

    “These reports indicate that an increase in DI is associated with an increase in the discrimination index.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Items constructed using AI had good psychometric properties and quality.

    “Items constructed using AI had good psychometric properties and quality, measuring higher-order domains.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Items constructed using AI measured higher-order domains.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    These discrepancies between the absolute difficulty index and rating could be due to differences in experts' opinions about items, experiences, and the process of estimating the difficulty of test items [59].

    “These discrepancies between the absolute difficulty index and rating could be due to differences in experts’ opinions about items, experiences, and the process of estimating the difficulty of test items [59].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] Hambleton RK, Jirka SJ. Anchor-based methods for judgmentally estimating item statistics. Handbook of test development. edn.: Routledge; 2011. pp. 413–34. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study did not test whether the AI tool would generate the same questions (items) on other occasions.

    “We did not test whether the AI tool would generate the same questions (items) on other occasions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study context was TBL.

    “The study context was TBL.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The number of items was small.

    “A small number of items.”
    Quote in the paperSubstantial claim
64 of 64

  • The paper relies on this without support.

    The AI tool ChatPdf.com is safe and maintains confidentiality.

    “5) safe tool, and 6) confidentiality.”

  • The paper relies on this without support.

    The sample of 48 students is sufficient for item analysis.

    “The total number of participants included in the study was 48.”

  • The paper relies on this without support.

    Difficult items with no flaws are answered by high-achieving rather than low-achieving students.

    “Difficult items with no flaws (content, technical, or grammatical flaws) were answered by good (high achievers) rather than low students (high achievers).”

  • The paper relies on this without support.

    The i-Rat scores reflect students' performance and competence.

    “Such clustering of students’ scores may indicate that students’ performance and competence are equal or that the items evaluate students’ knowledge well.”

  • The paper relies on this without support.

    Items constructed using AI measured higher-order domains.

    “Items constructed using AI had good psychometric properties and quality, measuring higher-order domains.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Three studies reported the use of AI, particularly ChatGPT, for generating exam questions.

    “Three studies reported the use of AI, particularly ChatGPT, for generating exam questions [133, 134, 136].”

    From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [133]

    The Methods section, “Generation of question (item),” reports using an AI tool to construct MCQs, so this paper supports its inclusion as a study of AI-generated exam questions. But “The selection of an AI tool” identifies ChatPdf.com, not ChatGPT; this source does not support the ChatGPT-specific part of the sentence. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • AI assisted students in conducting self-assessments by generating questions or identifying key points.

    “AI assisted students in conducting self-assessments by generating questions or identifying key points [133]”

    From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [133]

    The Methods sections on question generation and evaluation report that AI generated MCQs used in students’ individual readiness tests. The paper does not report students using AI to conduct self-assessments or to identify key points; those parts of the claim are unsupported. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The AI tool is named differently in the Discussion

    Major

    The Methods identify ChatPdf.com as the tool used, whereas the Discussion identifies ChatPDF.ai, which the Methods list as a separate candidate tool.

  • Discrimination range excludes an item with a zero index

    Minor

    The stated range ends at 0.15, but Table 2 reports a discrimination index of 0 for question 2.

  • Five of ten items is not more than half

    Minor

    The Discussion says more than half the items had excellent discrimination, but the Results report five of ten, exactly half.

  • Question 5 is classified as moderately difficult despite its index

    Minor

    Question 5 has a difficulty index of 37.5 but is labelled M.DI. Under the paper's stated cutoffs, it is difficult.

  • Question 2 is classified as moderately difficult despite its index

    Minor

    Question 2 has a difficulty index of 33.3 but is labelled M.DI. The paper defines indices below 39 as difficult.

  • Conflicting counts of non-functional distractors

    Minor

    The Results give both 26 and 4 as the number of non-functional distractors. The table's NFD counts total 4.

6 of 6

Cites

Flagged first, then by citations

Research works printed but not linked: 16

  1. Thorat S, Gupta M, Wasnik M. Item analysis–utility for increase in MCQ validity in biochemistry for MBBS students. J Educ Technol Health Sci. 2019;6(3):88–9.
  2. Khare AS, Kadam AA, Verma A, Avachar K. Analysis of Difficulty Index of single best response type of multiple choice questions in Physiology by Post Validation. Lat Am J Pharm. 2023;42(3):50–5.
  3. Zhai X, Chu X, Chai CS, Jong MSY, Istenic A, Spector M, Liu J-B, Yuan J, Li Y. A review of Artificial Intelligence (AI) in education from 2010 to 2020. Complexity. 2021;2021:1–18.
  4. González-Calatayud V, Prendes-Espinosa P, Roig-Vila R. Artificial intelligence for student assessment: a systematic review. Appl Sci. 2021;11(12):1–15. · DOI · Paper page
  5. Swiecki Z, Khosravi H, Chen G, Martinez-Maldonado R, Lodge JM, Milligan S, Selwyn N, Gašević D. Assessment in the age of artificial intelligence. Computers Education: Artif Intell. 2022;3:100075–85.
  6. Choi J. Automatic item generation with machine learning techniques. In: Application of Artificial Intelligence to Assessment edn. Edited by Jiao H, Cissitz R. USA: Information Age Publishing; 2020: 189–210.
  7. Case SM, Swanson DB. Writing one-best-answer questions for the Basic and Clinical sciences. Constructing written test questions for the Basic and Clinical sciences. 3 ed. Philadelphia: National Board of Medical Examiners; 2016. pp. 31–66.
  8. Rezigalla AA. Item analysis: Concept and application. In: Medical Education for the 21st Century edn. Edited by Firstenberg MS, Stawicki SP: IntechOpen; 2022: 105–120.
  9. Reinhardt BM. Factors Affecting Coefficient Alpha: A Mini Monte Carlo Study. In: The annual meeting of the SouthwestEducationea Research Association San Antonio: ERIC; 1991: 1–32.
  10. Bell BA. Pretest–Posttest Design. In: Encyclopedia of research design. Volume 2, edn. Edited by Salkind NJ. Thousand Oaks: SAGE Publications, Inc.; 2014: 1087–1092.
  11. Rezigalla AA. Item analysis: Concept and application. In: Medical Education for the 21st Century edn. Edited by Firstenberg MS, Stawicki SP. London: Intechopen; 2022: 1–16.
  12. Kumar D, Jaipurkar R, Shekhar A, Sikri G, Srinivas V. Item analysis of multiple choice questions: a quality assurance test for an assessment tool. Med J Armed Forces India. 2021;1(77):85–S89. · DOI · Paper page
  13. Hassan S, Hod R. Use of item analysis to improve the quality of single best answer multiple choice question in summative assessment of undergraduate medical students in Malaysia. Educ Med J 2017, 9(3). · DOI · Paper page
  14. Kehoe J. Basic item analysis for multiple-choice tests. Practical Assess Res Evaluation. 1994;4(10):1–3.
  15. Alareifi RM. Analysis of MCQs in summative exam in English: Difficulty Index, discrimination index and relationship between them. J Eduction Hum Sci. 2023;20:124–35.
  16. Soraya S, Shabani A, Kamalzadeh L, Kashaninasab F, Rashedi V, Saeidi M, Seddigh R, Asadi S. Predictability of discrimination coefficient and Difficulty Index of Psychiatry multiple-choice questions. J Iran Med Council. 2021;4(3):165–72.

Cited by

Sorted by citations

Related

Authors
Rezigalla, A. A.
Affiliations
University of Bisha (SA)
Corresponding author
Rezigalla, Assad Ali
ORCID
Rezigalla, Assad Ali
Venue
BMC Medical Education
Publisher
Springer Science and Business Media LLC
Published
2024-03-06
Volume
24
Issue
1
Pages
247-247
ISSN
1472-6920 (ISSN-L)
DOI
10.1186/s12909-024-05250-3
PMC
PMC10919037
OpenAlex
W4392501374
Semantic Scholar
777e53fe90cf797e7dae25c2114bf3866dc66760
Cited by
36
References
42
Citations by year
2026: 12 · 2025: 17 · 2024: 7
FWCI
3.37
Type
journal article
Language
en
Update notices
correction (2024-03-21)
Open access
Licence
Locations
  • BMC Medical Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
University of Bisha
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
artificial intelligence, automated item generation, multiple-choice questions, test item quality, item difficulty, item discrimination, distractor analysis, ChatPDF, formative assessment, test bank, medical education assessment
MeSH terms
Competency-Based Education, Cross-Sectional Studies, Education, Medical, Humans, Psychometrics, Research Design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review