Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 5

The DOI matches its Crossref record; the references it cites are not checked yet.

Are medical school preclinical tests biased for sex and race? A differential item functioning analysis

Dale, Esther Dasari, Abulela, Mohammed A. A., Jia, Hao, Violato, Claudio·2025·BMC Medical Education
DOI 10.1186/s12909-024-06540-6

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Background: A common practice in assessment development, fundamental for fairness and consequently the validity of test score interpretations and uses, is to ascertain whether test items function equally across test-taker groups. Accordingly, we conducted differential item functioning (DIF) analysis, a psychometric procedure for detecting potential item bias, for three preclinical medical school foundational courses based on students’ sex and race. Methods: The sample included 520, 519, and 344 medical students for anatomy, histology, and physiology, respectively, collected from 2018 to 2020. To conduct DIF analysis, we used the Wald test based on the two-parameter logistic model as utilized in the IRTPRO software. Results: The three assessments had as many as one-fifth of the items that functioned statistically differentially across one or more of the variables sex and race: 10 out of 49 items (20%), six out of 40 items (15%), 5 out of 45 items (11%) showed statistically significant DIF for Anatomy , Histology , and Physiology courses, respectively. Measurement specialists and subject matter experts independently reviewed the items to identify construct-irrelevant factors as potential sources for DIF as demonstrated in Appendix A. Most identified items were generally poorly written or had unclear images. Conclusions: The validity of score-based inferences, particularly for group comparisons, requires test items to function equally across test-taker groups. In the present study, we found DIF of some items for sex and race in three content areas. The present approach should be utilized in other medical schools to address the generalizability of the present findings. Item level DIF should also be routinely conducted as part of psychometric analyses for basic sciences courses and other assessments. Clinical trial number: Not applicable.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12909-024-06540-6

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 5

Research works printed but not linked: 38

  1. Abulela MAA, Rios JA. Comparing the robustness of three nonparametric DIF procedures to differential rapid guessing. Appl Meas Educ. 2022;35(2):81–94. · DOI
  2. Rubright J, Jodoin M, Woodward S, Barone M. Differential item functioning analysis of United States Medical licensing examination step 1 items. Acad Med. 2022;97(5):718–22. · DOI
  3. Ghaffari-Rafi A, Lee RE, Fang R, Miles JD. Multivariable analysis of factors associated with USMLE scores across U.S. medical schools. BMC Med Educ. 2019;19:154. · DOI
  4. Konopasek L, Norcini J, Krupat E. Focusing on the formative: building an assessment system aimed at student growth and development. Acad Med. 2016;91(11):1492–97. · DOI
  5. Van der Vleuten CPM, Swanson DB. Assessment of clinical skills with standardized patients: state of the art revisited. Teach Learn Med. 2013;25(1):17–25.
  6. Baig L, Violato C, Crutcher R. A construct validity study of clinical competence: a multitrait multimethod matrix approach. J Contin Educ Health Prof. 2010;30(1):19–25. · DOI
  7. Ratanawongsa N, Thomas PA, Marinopoulos SS, Dorman T, Wilson LM, Ashar BH, Magaziner JL, Miller RG, Prokopowicz GP, Qayyum R, Bass EB. The reported validity and reliability of methods for evaluating continuing medical education: a systematic review. Acad Med. 2008;83(3):274–83. · DOI
  8. Turner JL, Dankoski ME. Objective structured clinical exams: a critical review. Fam Med. 2008;40(8):574–8.
  9. Walker CM. What’s the DIF? Why differential item functioning analyses are an important part of instrument development and validation. J Psychoeduc Assess. 2011;29(4):364–76. · DOI
  10. Dixon D. Gender differences in academic qualifications and medical school performance of osteopathic medical students. Med Sci Educ. 2007;17(1):33–7.
  11. Haist SA, Wilson JF, Elam CL, Blue AV, Fosson SE. The Effect of gender and age on medical school performance: an important interaction. Adv Health Sci Educ Theory Pract. 2000;5(3):197–205. · DOI
  12. Nuzhat A, Salem RO, Hamdan NA, Ashour N. Gender differences in learning styles and academic performance of medical students in Saudi Arabia. Med Teach. 2013;35(Suppl 1):S78–82. · DOI
  13. Davis D, Dorsey JK, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Acad Med. 2013;88(5):593–602. · DOI
  14. Woolf K, Potts HWW, McManus IC. Ethnicity and academic performance in UK trained doctors and medical students: systematic review and meta-analysis. BMJ. 2011;342:d901. · DOI
  15. Isik U, Wouters A, Ter Wee MM, Croiset G, Kusurkar RA. Motivation and academic performance of medical students from ethnic minorities and majority: a comparative study. BMC Med Educ. 2017;17:233. · DOI
  16. Miller GD, Kemmelmeier M, Dupey P. Gender differences in worry during medical school. Med Educ. 2013;47(9):932–41. · DOI
  17. Roter DL, Hall JA, Aoki Y. Physician gender effects in medical communication: a meta-analytic review. JAMA. 2002;288(6):756–64. · DOI
  18. Cooper-Patrick L, Gallo JJ, Gonzales JJ, Vu HT, Powe NR, Nelson C, Ford DE. Race, gender, and partnership in the patient-physician relationship. JAMA. 1999; 11;282(6):583-9. · DOI
  19. Wright CD, Eddy SL, Wenderoth MP, Abshire E, Blankenbiller M, Brownell SE. Cognitive difficulty and format of exams predicts gender and socioeconomic gaps in exam performance of students in introductory biology courses. CBE Life Sci Educ. 2016;15(2):ar23. · DOI
  20. Ackerman TA. A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. J Educ Meas. 1992;29(1):67–91. · DOI
  21. Roussos L, Stout W. A multidimensionality-based DIF analysis paradigm. Appl Psychol Meas. 1996;20(4):355–71. · DOI
  22. Holland PW, Wainer H. Differential item functioning. New York: Routledge; 2012. · DOI
  23. Zumbo BD. Three generations of DIF analyses: considering where it has been, where it is now, and where it is going. Lang Assess Q. 2007;4(2):223–33. · DOI
  24. Magis D, Béland S, Tuerlinckx F, de Boeck PA. A general framework and an R package for the detection of dichotomous differential item functioning. Behav Res Methods. 2010;42(3):847–62. · DOI
  25. Martinková P, Drabinová A, Liaw Y-L, Sanders EA, McFarland JL, Price RM. Checking equity: why differential item functioning analysis should be a routine part of developing conceptual assessments. CBE Life Sci Educ. 2017;16(2):rm2. · DOI
  26. Bielinski J, Davison ML. A sex difference by item difficulty interaction in multiple-choice mathematics items administered to national probability samples. J Educ Meas. 2001;38(1):51–77. · DOI
  27. Shanmugam SKSSN. Gender-related differential item functioning of mathematics computation items among non-native speakers of English. Math Enthus. 2020;17(1):6. · DOI
  28. Hyde JS, Fennema E, Ryan M, Frost LA, Hopp C. Gender comparisons of mathematics attitudes and affect: a meta-analysis. Psychol Women Q. 1990;14(3):299–324. · DOI
  29. Nuttall DL, Goldstein H, Prosser R, Rasbash J. Differential school effectiveness. Int J Educ Res. 1989;13(7):769–76. · DOI
  30. Dyrbye LN, Thomas MR, Eacker A, Harper W, Massie FS Jr, Power DV, Huschka M, Novotny PJ, Sloan JA, Shanafelt TD. Race, ethnicity, and medical student well-being in the United States. Arch Intern Med. 2007;167(19):2103–9. · DOI
  31. Cai L, Thissen D, du Toit S. H. C. IRTPRO for windows (Version 5). Lincolnwood: Scientific Software International; 2020.
  32. Cai L. SEM of another flavour: two new applications of the supplemented EM algorithm. Br J Math Stat Psychol. 2008;61(Pt 2):309–29. · DOI
  33. Sireci SG, Rios JR. Decisions that make a difference in detecting differential item functioning. Educ Res Eval. 2013;19(2–3):170–87. · DOI
  34. Sireci SG, Mullane LA. Evaluating test fairness in licensure testing: the sensitivity review process. CLEAR exam rev. 1994;5(2):22–8.
  35. Maccoby EE, Jacklin CN. The psychology of sex differences. Stanford University Press; 1974. · DOI · Paper page
  36. Maeda Y, Yoon SY. A meta-analysis on gender differences in mental rotation ability measured by the Purdue spatial visualization tests: visualization of rotations (PSVT:R). Educ Psychol Rev. 2013;25(1):69–94. · DOI
  37. Yammine K, Violato C. A meta-analysis of the educational effectiveness of three-dimensional visualization technologies in teaching anatomy. Anat Sci Educ. 2015;8(6):525–38. · DOI
  38. Donnon T, DesCôteaux JG, Violato C. Impact of cognitive imaging and sex differences on the development of laparoscopic suturing skills. Can J Surg. 2005;48(5):387–93.

Metadata

Authors
Dale, E. D., Abulela, M. A. A., Jia, H., & Violato, C.
Venue
BMC Medical Education
Publisher
Springer Science and Business Media LLC
Published
2025-01-29
Volume
25
Issue
1
ISSN
1472-6920
DOI
10.1186/s12909-024-06540-6
Cited by
5
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Are medical school preclinical tests biased for sex and race? A differential item functioning analysis. https://app.cochat.ai/papers/10.1186/s12909-024-06540-6
The paper
Dale, E. D., Abulela, M. A. A., Jia, H., & Violato, C. (2025). Are medical school preclinical tests biased for sex and race? A differential item functioning analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-06540-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.