CoChat
Sign Up
Cited by 5Psychometric Methodologies and TestingInnovations in Medical EducationMedical Education and Admissions

Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations

Open access
Xu, Lingling, Jiang, Zhehan, Han, Yuting, Liang, Haiying, Ouyang, Jinying·2023·Perspectives on Medical Education
DOI 10.5334/pme.855PMC PMC10624130
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction: The accurate assessment of health professionals’ competence is critical for ensuring public health safety and quality of care. Computerized Adaptive Testing (CAT) based on the Item Response Theory (IRT) has the potential to improve measurement accuracy and reduce respondent burden. In this study, we conducted psychometric simulations to develop a CAT for evaluating the candidates’ competence of health professionals. Methods: The initial CAT item bank was sourced from the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test in China, consisting of 300 multiple-choice items. We randomly selected response data from 2000 Chinese clinical medicine undergraduates for analysis. Two types of analyses were performed: first, evaluating the psychometric properties of all items to meet the requirements of CAT; and second, conducting multiple CAT simulations using both simulated and real response data. Results: The final CAT item bank consisted of 121 items, for which item parameters were calculated using a two-parameter logistic model (2PLM). The CAT simulations, based on both simulated and real data, revealed sufficient marginal reliability (coefficient of marginal reliability above 0.750) and criterion-related validity (Pearson’s correlations between CAT scores and aggregate scores of the SCTCMU exceeding 0.850). Discussion: In national-level medical education assessment, there is an increasing need for concise yet valid evaluations of candidates’ competence of health professionals. The CAT developed in this study demonstrated satisfactory reliability and validity, offering a more efficient assessment of candidates’ competence of health professionals. The psychometric properties of the CAT could lead to shorter test durations, reduced information loss, and a decreased testing burden for participants.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.5334/pme.855

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • By selecting items that maximize information about the current ability estimate, CAT achieves equivalent precision with far fewer questions.

    “By selecting items that maximize information about the current ability estimate, CAT achieves equivalent precision with far fewer questions15.”

    From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [15]

    Methods (CAT Simulation; page not provided) says the maximum Fisher information method selects the item with the highest information at the current estimated ability level. Discussion (page not provided) reports high correlations with full-assessment scores using substantially fewer items, but says measurement accuracy decreased when test length was reduced; it does not establish equivalent precision. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Computerized adaptive testing is the gold standard in high-stakes human assessments such as nursing and medical licensure exams.

    “Computerized Adaptive Testing (CAT), built upon IRT, is the gold standard in high-stakes human assessments (e.g., nursing and medical licensure exams).”

    From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [15]

    The Introduction describes CAT as a promising method that has received considerable attention in medical education, and the paper identifies it as based on item response theory. It does not call CAT the gold standard or establish its use in nursing licensure exams. The Introduction mentions medical licensing exams as examples of standardized testing, not as evidence that CAT is the standard for them. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Simulating 3600 examinees with true ability parameters drawn from a uniform distribution across 36 discrete ability levels from −3.5 to +3.5 provides a more stringent test of the CAT algorithm's robustness across the entire ability spectrum.

    “We simulated 3600 examinees with true ability parameters drawn from a uniform distribution, spanning 36 discrete ability levels from −3.5 to +3.5 at intervals of 0.2. Each ability level was replicated 100 times. This uniform sampling, as opposed to a normal distribution, provides a more stringent test of the CAT algorithm’s robustness across the entire ability spectrum15,29,33.”

    From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [15]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Randomly selecting the first item from the full bank ensures an unbiased starting point.

    “the CAT was initiated by randomly selecting a first item from the full bank, ensuring an unbiased starting point15,29.”

    From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [15]

    The Methods > CAT Simulation passage says the CAT starts by selecting a single item. It does not say that the item is selected randomly from the full bank or that this ensures an unbiased starting point. Page numbers are unavailable in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [14]Flens G, Smits N, Carlier I, van Hemert AM, de Beurs E. Simulating computer adaptive testing with the Mood and Anxiety Symptom Questionnaire. Psychol Assess. 2016; 28(8): 953– 62. DOI: 10.1037/pas0000240 26691506 · DOI
  2. [30]Birnbaum A. On the estimation of mental ability. Series Rep. 1958; 15: 7755– 7723.

Cited by

Sorted by citations

Related

Authors
Xu, L., Jiang, Z., Han, Y., Liang, H., & Ouyang, J.
Affiliations
  • Beijing Language and Culture University (CN)
  • University College London (GB)
  • Peking University (CN)
Corresponding author
  • Jiang, Zhehan
  • Han, Yuting
  • Liang, Haiying
  • Ouyang, Jinying
ORCID
Venue
Perspectives on Medical Education
Publisher
Ubiquity Press, Ltd.
Published
2023-10-31
Volume
12
Issue
1
Pages
462-471
ISSN
  • 2212-2761 (ISSN-L)
  • 2212-277X
DOI
10.5334/pme.855
PMC
PMC10624130
OpenAlex
W4388043027
Cited by
5
References
26
Citations by year
2026: 2 · 2025: 2 · 2024: 1
FWCI
0.92
Type
journal article
Language
en
Open access
Licence
Locations
  • Perspectives on Medical Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Natural Science Foundation of China (72104006)
  • Peking University Health Science Center (BMU2021YJ010)
  • Key Laboratory of Digital Educational Publishing Technology (RJB0623001)
  • Standards and Digital Education Research Institute of the Peoples Education Press (RJB0623001)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
Keywords
computerized adaptive testing, item response theory, item bank development, medical competence assessment, criterion validity, two-parameter logistic model, multiple-choice questions, test length reduction
MeSH terms
Computerized Adaptive Testing, Health Personnel, Humans, Psychometrics, Students, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review