Improving the Quality of MCQs by Enhancing Cognitive Level and using Psychometric Analysis
Open accessAbstract
Faculty development programs are an essential part of any reputable institution. Faculty training through various conferences and workshops will help develop competencies for high quality assessment. Objective: To evaluate the effectiveness of ongoing faculty training initiatives aimed at improving the standard of MCQ development. Methods: Faculty members of FRPMC were engaged in this observational, quasi-experimental study. Convenient sampling was done. Three different questionnaires were provided to the participants. The participants were tasked with accurately identifying flaws, cognitive levels in the pre- and post-test questionnaire items, as well as post hoc analysis to discard or modify the Items. Results: Items with multiple flaws were assessed with mean value in pre-test score was 1.349 compared to post-test score 3.442 which were statistically significant (p-value <0.05). The pre- and post-test questionnaire to correctly identify the cognitive levels showed pre-test 77.5% compared to 87.5% participants identified correct cognitive level in post-test. In post hoc analysis, participants were able to identify 5 questions out of 7 which need to be discarded and 3 questions out of 5 which needed improvement according to the key given to them. Conclusions: Our study revealed that workshop helped faculty identify item flaws with significant improvement in pre- and post-test scores. The cognitive capabilities of faculty were improved in all three levels of cognition, with greatest improvements shown in higher complexity questions (C3). There was a noticeable improvement in knowledge and the participant's capacity to identify and correct errors.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The questionnaire to subject experts was based on and adapted from the work of Susan and David (2016) and other authors.
“The questionnaire to the subject experts about item construction and quality was based on and adapted from the work of Susan and David (2016) and other authors [2, 4, 32] (Table 1).”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [4]
The construction of high-quality MCQs type A has been reported to be difficult and time-consuming.
“The construction of high-quality MCQs type A has been reported to be difficult and time-consuming [4, 7–9].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Core principles of assessment in competency-based medical education.
Lockyer, J., Carraccio, C., Chan, M.-K., Hart, D., Smee, S., Touchie, C., Holmboe, E. S., Frank, J. R., & Collaborators, O. B. O. T. I. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1315082 · Cited by 493 · cited 2× · see citing works
“The main purpose of assessment is to provide feedback on the process of learning [1].”
BackgroundA history of assessment in medical education
Schuwirth, L., & van der Vleuten, C. P. M. (2020) · Advances in Health Sciences Education · doi:10.1007/s10459-020-10003-0 · Cited by 149 · see citing works
Rush, B. R., Rankin, D. C., & White, B. J. (2016) · BMC Medical Education · doi:10.1186/s12909-016-0773-3 · Cited by 126 · see citing works
Faculty development programs improve the quality of Multiple Choice Questions items' writing
Abdulghani, H. M., Ahmad, F., Irshad, M., Khalil, M. S., Al-Shaikh, G. K., Syed, S., Al-Drees, A. A., Al-Rowais, N. A., & Haque, S. (2015) · Scientific Reports · doi:10.1038/srep09556 · Cited by 110 · see citing works
Climbing Bloom's taxonomy pyramid: Lessons from a graduate histology course
Zaidi, N. L. B., Hwang, C. D., Scott, S., Stallard, S., Purkiss, J. A., & Hortsch, M. (2017) · Anatomical Sciences Education · doi:10.1002/ase.1685 · Cited by 75 · see citing works
Examining Bloom’s Taxonomy in Multiple Choice Questions: Students’ Approach to Questions
Stringer, J. K., Santen, S. A., Lee, E., Rawls, M., Bailey, J. M., Richards, A., Perera, R. A., & Biskobing, D. M. (2021) · Medical Science Educator · doi:10.1007/s40670-021-01305-y · Cited by 72 · see citing works
Medical students create multiple-choice questions for learning in pathology education: a pilot study
Grainger, R., Dai, W., Osborne, E., & Kenwright, D. N. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1312-1 · Cited by 62 · see citing works
Abdulghani, H. M., Irshad, M., Haque, S., Ahmad, T., Sattar, K., & Khalil, M. S. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185895 · Cited by 51 · see citing works
Ali, S. H., Carr, P. A., & Ruit, K. G. (2016) · Journal of the Scholarship of Teaching and Learning · doi:10.14434/josotl.v16i1.19106 · Cited by 49 · see citing works
Holzinger, A., Lettner, S., Steiner‐Hofbauer, V., & Melser, M. C. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02239-6 · Cited by 47 · see citing works
Dellinges, M. A., & Curtis, D. A. (2017) · Journal of Dental Education · doi:10.21815/jde.017.047 · Cited by 37 · see citing works
Effect of faculty training on quality of multiple-choice questions
Gupta, P., Meena, P., Khan, A. M., et al. (2020) · International Journal of Applied and Basic Medical Research · doi:10.4103/ijabmr.ijabmr_30_20 · Cited by 36 · see citing works
A One‐Day Dental Faculty Workshop in Writing Multiple‐Choice Questions: An Impact Evaluation
AlFaris, E., Naeem, N., Irfan, F., et al. (2015) · Journal of Dental Education · doi:10.1002/j.0022-0337.2015.79.11.tb06026.x · Cited by 30 · see citing works
Evaluating the Quality of Multiple Choice Question in Paediatric Dentistry Postgraduate Examinations
Kowash, M., Hussein, I., & Halabi, M. A. (2019) · Sultan Qaboos University medical journal · doi:10.18295/squmj.2019.19.02.009 · Cited by 20 · see citing works
Nonfunctional distractor analysis: An indicator for quality of Multiple choice questions
Sajjad, M., Iltaf, S., & Khan, R. A. (2020) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.36.5.2439 · Cited by 20 · see citing works
The impact of a faculty development seminar on the quality of multiple-choice questions
Nemec, E. C., & Welch, B. E. (2016) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2015.12.008 · Cited by 15 · see citing works
Farooqui, F., Saeed, N., Aaraj, S., et al. (2018) · Cureus · doi:10.7759/cureus.3773 · Cited by 14 · see citing works
Psychometric analysis of Anatomy MCQs in Modular examination
Islam, Z. U., & Usmani, A. (2017) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.335.12382 · Cited by 13 · see citing works
Construction of Multiple Choice Questions Before and After An Educational Intervention
Tenzin, K., Dorji, T., & Tenzin, T. (2017) · Journal of Nepal Medical Association · doi:10.31729/jnma.2976 · Cited by 12 · see citing works
Balaha, M. H., El-Ibiary, M. T., El-Dorf, A., El-Shewaikh, S. L., & Balaha, H. M. (2022) · Avicenna Journal of Medicine · doi:10.1055/s-0042-1755332 · Cited by 11 · see citing works
Arooj, M., Mukhtar, K., Khan, R. A., et al. (2021) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.37.2.3475 · Cited by 10 · see citing works
Faculty perceptions on one‐best MCQ development
Adnan, S., Sarfaraz, S., Nisar, M. K., et al. (2022) · The Clinical Teacher · doi:10.1111/tct.13529 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Pickford JC and Newcomb LH. Relationship of cognitive level of instruction to students' cognitive level of achievement. NACTA Journal. 1989 Jun; 33(2): 56-9.
- Khan MU and Aljarallah BM. Evaluation of Modi ed Essay Questions (MEQ) and Multiple Choice Questions (MCQ) as a tool for Assessing the Cognitive Skills of Undergraduate Medical Students. International Journal of Health Sciences. 2011 Jan; 5(1): 39-43.
- Holzinger A, Lettner S, Steiner-Hofbauer V, Capan Melser M. How to assess? Perceptions and preferences of undergraduate medical students concerning traditional assessment methods. BMC Medical Education. 2020 Sep; 20(1): 312. doi:
- Hingorjo MR and Jaleel F. Analysis of one-best MCQs: the di culty index, discrimination index and distractor e ciency. Journal of Pakistan Medical Association. 2012 Feb; 62(2): 142-7.
- Gupta P, Meena P, Khan AM, Malhotra RK, Singh T. Effect of faculty training on quality of multiple- choice questions. International Journal of Applied [17]
Cited by
Sorted by citations
AI in medical education: uses of AI in construction type A MCQs
Rezigalla, A. A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05250-3 · Cited by 36 · see citing works
- Authors
- Rahim, M. F., Qassim Bham, S., Khan, S., Ansari, T., & Ahmed, M.
- Affiliations
- Air University (PK)
- Venue
- Pakistan Journal of Health Sciences
- Publisher
- CrossLinks International Publishers
- Published
- 2023-04-30
- Pages
- 115-121
- ISSN
- 2790-9344 (ISSN-L)
- 2790-9352
- OpenAlex
- W4375853313
- Cited by
- 3
- References
- 22
- Citations by year
- 2025: 2 · 2024: 1
- FWCI
- 0.77
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Pakistan Journal of Health Sciences (publishedVersion, cc-by)
- Greater South Information System
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Education › Educational Assessment and Pedagogy
- Social Sciences › Education › Health Education and Validation
- Keywords
- faculty development, cognitive levels, psychometric assessment, item analysis, post hoc analysis, workshop effectiveness, medical education assessment, university faculty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
