Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking

Classified as probably not a substantial claim. May combine two claims.

Recent studies caution that medical MCQ performance can be inflated by test-taking heuristics and may not reliably indicate clinically meaningful understanding or readiness.

“Recent studies caution that medical MCQ performance can be inflated by test-taking heuristics and may not reliably indicate clinically meaningful understanding or readiness20,21.”

• cites [20] Griot, M., Vanderdonckt, J. & Yuksel, D. Implementation of large language models in electronic health records. PLOS Digit. Health 4, e0001141 (2025). sourcePartly supported

  The Introduction says that the relevance of MCQs for evaluating LLMs is debated and that strong benchmark results are difficult to translate into real clinical practice. It does not say that test-taking heuristics inflate MCQ scores or specifically assess whether those scores demonstrate clinically meaningful understanding. No page number is provided in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [21] Gu, Y. et al. The illusion of readiness in health AI. Preprint at https://doi.org/10.48550/arXiv.2509.18234 (2025). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Computerized adaptive testing is the gold standard in high-stakes human assessments such as nursing and medical licensure exams.

“Computerized Adaptive Testing (CAT), built upon IRT, is the gold standard in high-stakes human assessments (e.g., nursing and medical licensure exams).”

• cites [15] Xu, L., Jiang, Z., Han, Y., Liang, H. & Ouyang, J. Developing computerized adaptive testing for a national health professionals exam: an attempt from psychometric simulations. Perspect. Med. Educ. 12, 462–471 (2023). sourcePartly supported

  The Introduction describes CAT as a promising method that has received considerable attention in medical education, and the paper identifies it as based on item response theory. It does not call CAT the gold standard or establish its use in nursing licensure exams. The Introduction mentions medical licensing exams as examples of standardized testing, not as evidence that CAT is the standard for them. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Projecting to a large benchmark scenario (Med-HALT; 59,254 items), the cost of a full-length evaluation would be on the order of $1475 under stated pricing assumptions, whereas CAT-based evaluation would be under $5.

“Using empirically observed testing characteristics (e.g., an average of 37 items per administration and per-item token usage) and projecting them to a large benchmark scenario (Med-HALT; 59,254 items10), we estimate that the cost of a full-length evaluation would be on the order of $1475 under the stated pricing assumptions, whereas the CAT-based evaluation would be under $5.”

• cites [10] Pal, A., Umapathi, L. K. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. Conf. Comput. Nat. Lang. Learn. 314–334 (2023). sourcePartly supported

  Section 2, “Datasets Statistics” (page unavailable), identifies Med-HALT as a large benchmark and reports 18,866 samples per reasoning-hallucination task plus a separate PubMed portion. It does not give the cited figure of 59,254 items. The $1,475 and under-$5 costs are explicitly the citing authors’ estimates, not claims attributed to Med-HALT. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: At current pricing, a single full-benchmark evaluation can exceed $1000.

May combine two claims.

LLMs' medical performance can improve by nearly 20% within a single year, necessitating timely and repeated evaluation.

“This becomes particularly problematic given that LLMs’ medical performance can improve by nearly 20% within a single year11, necessitating timely and repeated evaluation to accurately track progress—yet the substantial financial and operational burden makes such monitoring difficult to sustain.”

• cites [11] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). sourcePartly supported

  The 'State of the art on MedQA' section reports that Flan-PaLM exceeded the previous best MedQA accuracy by 17.3 percentage points, supporting the nearly 20% comparison for that benchmark. It does not establish that this improvement occurred within a single year. The call for repeated evaluation is the citing authors’ inference, not a finding attributed to this source. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Real-world deployments of clinical LLM assistants in electronic health records contexts emphasize summarization, information retrieval, and documentation support rather than autonomous clinical decision-making.

“This positioning also aligns with observed real-world deployments of clinical LLM assistants in electronic health records contexts, where primary use cases emphasize summarization, information retrieval, and documentation support rather than autonomous clinical decision-making22.”

• cites [22] Griot, M., Vanderdonckt, J., Yuksel, D. & Hemptinne, C. Pattern recognition or medical knowledge? The problem with multiple-choice questions in medicine. Proc. Annu. Meet. Assoc. Comput. Linguist. 5321–5341 (2025). sourceCould not verify

  The Medical Implications section discusses risks of clinical LLM use and clinical decision support, but does not describe observed deployments in electronic health records or their primary use cases. Searches for EHR summarization, information retrieval, and documentation support found no passage addressing that comparison. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Randomly selecting the first item from the full bank ensures an unbiased starting point.

“the CAT was initiated by randomly selecting a first item from the full bank, ensuring an unbiased starting point15,29.”

• cites [15] Xu, L., Jiang, Z., Han, Y., Liang, H. & Ouyang, J. Developing computerized adaptive testing for a national health professionals exam: an attempt from psychometric simulations. Perspect. Med. Educ. 12, 462–471 (2023). sourcePartly supported

  The Methods > CAT Simulation passage says the CAT starts by selecting a single item. It does not say that the item is selected randomly from the full bank or that this ensures an unbiased starting point. Page numbers are unavailable in our copy. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [29] Wu, Y., Cai, Y. & Tu, D. A computerized adaptive testing advancing the measurement of subjective well-being. J. Pac. Rim Psychol. 13, e6 (2019). sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

CAT-based θ is not interpreted as clinical ability; instead, it is an operational proficiency index on a calibrated, secure knowledge bank under a fixed evaluation protocol.

“At the same time, consistent with this literature, we do not interpret CAT-based as clinical ability; instead, is an operational proficiency index on a calibrated, secure knowledge bank under a fixed evaluation protocol21.”

• cites [21] Gu, Y. et al. The illusion of readiness in health AI. Preprint at https://doi.org/10.48550/arXiv.2509.18234 (2025). sourcePartly supported

  The Discussion (p. 12) cautions that benchmark success does not establish medical viability, supporting the decision not to equate a test score with clinical ability. But the paper does not address CAT-based θ or define it as an operational proficiency index on a calibrated, secure knowledge bank under a fixed protocol; searches for those terms found no matching passage. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

Under a standardized protocol, the optimal CAT reduced evaluation cost by approximately 98% and preserved the full-bank rank-ordering in the evaluated set.

“Under a standardized protocol, our optimal CAT reduced evaluation cost by approximately 98% and preserved the full-bank rank-ordering in our evaluated set.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The random selection baseline failed to preserve the ranking of state-of-the-art models; the true top-performing model (o1) was incorrectly ranked as 3rd.

“Most critically, RS sampling failed to preserve the ranking of state-of-the-art models. As highlighted in the figure, the true top-performing model (o1) was incorrectly ranked as 3rd by the RS baseline, falling behind Gemini-pro and Gemini-2.5-pro due to the lack of challenging items.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The correlation with true ability scores remained high across all MFI conditions; even the SE ≤ 0.5 condition achieved a correlation of 0.96.

“The correlation with true ability scores remained high across all MFI conditions (Fig. 1, Panel C); even the highly efficient SE ≤ 0.5 condition achieved a correlation of 0.96, offering an optimal balance between precision and cost.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The CAT protocol required an average of only 37 items to meet the high-reliability stopping criterion, representing an average test length reduction of 98.7% compared to the 2815-item full bank.

“As shown in Fig. 5, Panel A, the CAT protocol required an average of only 37 items to meet the high-reliability stopping criterion. Compared to the 2815 items in the full bank, this represents an average test length reduction of 98.7%.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

Under fixed-length conditions, the MFI adaptive approach consistently minimized measurement error more effectively than random selection; at a test length of 50 items, MFI achieved an RMSE of 0.3982 versus 0.5061 for the random selection baseline.

“Under fixed-length conditions, the adaptive approach consistently minimized measurement error more effectively than RS; for instance, at a test length of 50 items, MFI achieved an RMSE of 0.3982, significantly lower than the 0.5061 observed for the RS baseline (Fig. 1, Panel B).”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The SE ≤ 0.316 condition required an average of only 58.3 items, a 97.9% reduction in test length compared to the full bank.

“Notably, the variable-length rules were extremely efficient: the SE ≤ 0.316 condition required an average of only 58.3 items, a 97.9% reduction in test length compared to the full bank.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The SE ≤ 0.316 rule achieves a positive Bias Improvement Rate, producing a more accurate, less systematically skewed estimate than even the full-bank test.

“Second, it is one of the few conditions to achieve a positive Bias Improvement Rate (Fig. 2, Panel A), demonstrating its unique strength in producing a more accurate, less systematically skewed estimate than even the full-bank test.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

The quote only partly states this claim.

The CAT theta estimates exhibited an exceptionally strong correlation with full-bank theta estimates, achieving r = 0.988.

“The exhibited an exceptionally strong correlation with , achieving r = 0.988.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The full item bank administration (2815 items) achieved the lowest RMSE of 0.100 and a near-perfect correlation of 0.9995 with true ability scores, with bias -0.0014.

“It established the benchmark for overall accuracy by achieving the lowest RMSE of 0.100 and a near-perfect correlation of 0.9995 with the true ability scores.”

• Also stated in: § Performance of stopping rules in the simulation study · plain

  “However, its bias was not the lowest, recorded at -0.0014 (Fig. 1, Panel A).”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

Local independence was supported: no item pairs had residual correlations exceeding |0.20| or LD statistics above |5|, suggesting minimal evidence of local dependence and negligible inflation of reliability due to residual item dependencies.

“No item pairs exhibited residual correlations exceeding |0.20 | , and no item pairs showed LD statistics above |5 | , suggesting minimal evidence of local dependence and that any potential inflation of reliability due to residual item dependencies is likely negligible.”

May combine two claims.

Discrimination parameters indicated high item quality (M = 1.01, SD = 0.08), with values ranging from 0.44 to 1.52.

“Discrimination parameters indicated high item quality (M = 1.01, SD = 0.08), with values ranging from 0.44 to 1.52.”

May combine two claims.

Difficulty parameters were well-centered around the mean ability of the human reference cohort (M = −0.01, SD = 0.20), spanning −1.11 to 1.44, supporting precise measurement around the ability range covered by the item bank.

“difficulty parameters were well-centered around the mean ability of the human reference cohort (M = −0.01, SD = 0.20), spanning a range from −1.11 to 1.44. This supports precise measurement around the ability range covered by the item bank.”

May combine two claims.

CAT can be operationalized to evaluate LLMs on a nationally calibrated, secure medical item bank with substantial efficiency gains while preserving comparative fidelity.

“This study shows that CAT can be operationalized to evaluate LLMs on a nationally calibrated, secure medical item bank with substantial efficiency gains while preserving comparative fidelity.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

The evaluation relies on MCQs, which primarily operationalize standardized medical knowledge in an exam-like format and may not fully capture real-world clinical reasoning.

“Our evaluation relies on MCQs, which primarily operationalize standardized medical knowledge in an exam-like format and may not fully capture real-world clinical reasoning.”

Classified as probably not a substantial claim.

Item parameters estimated from LLM responses highly correlated with those from expert predictions (Spearman's > 0.8).

“Another study found that item parameters estimated from LLM responses highly correlated with those from expert predictions (Spearman’s > 0.8)17.”

• cites [17] Zhuang, Y. et al. Survey of computerized adaptive testing: a machine learning perspective. IEEE Trans. Pattern Anal. Mach. Intell. (2026). sourceNot checked — no full text of the source

May combine two claims.

The observed estimation noise was insufficient to disrupt the relative ordering of the models, resulting in a robust leaderboard replication.

“Consequently, the observed estimation noise was insufficient to disrupt the relative ordering of the models, resulting in a robust leaderboard replication.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

Classified as probably not a substantial claim.

The adaptive methodology is not a substitute for real-world clinical validation or safety-oriented prospective studies.

“Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.”

May combine two claims.

The CAT strategy enables a level of measurement fidelity previously achievable only through exhaustive, cost-prohibitive methods, but at a mere 1.3% of the item cost and 2% of the computational and temporal cost.

“It enables a level of measurement fidelity previously achievable only through exhaustive, cost-prohibitive methods, but at a mere 1.3% of the item cost and 2% of the computational and temporal cost.”

• AssumptionRests on an assumption: The secure, non-public national medical item bank is calibrated on real human examinees.
• AssumptionRests on an assumption: The secure, non-public national medical item bank eliminates data contamination for LLM evaluation.

May combine two claims.

IRT models the probability of a correct response as a function of an examinee's latent proficiency and item characteristics, independent of the specific test-taker sample, unlike Classical Test Theory, which yields sample-dependent scores.

“Item Response Theory (IRT), regarded as a modern test theory, models the probability of a correct response as a function of an examinee’s latent proficiency (θ) and item characteristics, independent of the specific test-taker sample—unlike Classical Test Theory, which yields sample-dependent scores24,25.”

• cites [24] Embretson, S. E. & Reise, S. P. Item response theory for psychologists. (Lawrence Erlbaum Associates, 2000). sourceNot checked — no full text of the source
• cites [25] Hambleton, R. K. & Swaminathan, H. Estimation of ability. in Item Response Theory: Principles and Applications 85–106 (Springer, 1985). sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

For each LLM, θ is interpreted operationally as a proficiency index on the calibrated item bank under a standardized evaluation protocol, rather than as evidence of a human-like latent cognitive trait.

“For each LLM, is interpreted operationally as a proficiency index on this calibrated item bank under a standardized evaluation protocol, rather than as evidence of a human-like latent cognitive trait.”

• AssumptionRests on an assumption: LLMs are not human examinees and do not imply a single psychological latent trait.

May combine two claims.

Dichotomously scored MCQs support objective scoring and statistically stable parameter estimation relative to rubric-scored open-ended responses, where scoring subjectivity and rater effects can introduce additional noise and threaten parameter stability.

“Dichotomously scored MCQs support objective scoring and statistically stable parameter estimation relative to rubric-scored open-ended responses, where scoring subjectivity and rater effects can introduce additional noise and threaten parameter stability.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The human calibration sample defines the scale and item parameters under standard assumptions, supported by essential unidimensionality and minimal local dependence.

“The human calibration sample defines the scale and item parameters under standard assumptions (supported by essential unidimensionality and minimal local dependence)”

May combine two claims.

The results establish a practical and cost-efficient pathway for adaptive, psychometrically grounded benchmarking of standardized medical knowledge in LLMs, while clarifying the boundaries of what such scores can and cannot support in safety-critical contexts.

“Overall, our results establish a practical and cost-efficient pathway for adaptive, psychometrically grounded benchmarking of standardized medical knowledge in LLMs, while clarifying the boundaries of what such scores can and cannot support in safety-critical contexts.”

May combine two claims.

All major licensing exams use MCQs as their primary format, and leading medical LLM benchmarks also rely on MCQs.

“All major licensing exams use MCQs as their primary format: USMLE Step 1/2CK (USA), PLAB 1 (UK), MCCQE (Canada), AMC CAT (Australia), and China’s National Competency Test for Clinical Medicine Undergraduates. Leading medical LLM benchmarks also rely on MCQs: MedQA2, MMLU-Med3, MedMCQA9, and MedBench4.”

• cites MedQA2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites MMLU-Med3Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites MedMCQA9Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The two-parameter logistic (2PL) model is widely used in well-known assessments worldwide, such as TIMSS and PISA, for dichotomously scored items.

“We adopted the two-parameter logistic (2PL) model, which is widely used in well-known assessments worldwide (e.g., Trends in International Mathematics and Science Study, TIMSS26; Program for International Student Assessment, PISA27), for dichotomously scored items.”

• cites TIMSS26Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites PISA27Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Context-induced dependencies were minimized by using independent items and temperature = 0.

“We further minimized context-induced dependencies (independent items, temperature = 0)”

May combine two claims.

The CAT protocol depends only on calibrated item parameters, not response format, and could in principle be applied to any scored item type once a sufficiently large, human-calibrated bank becomes available.

“Our CAT protocol depends only on calibrated item parameters—not response format. In principle, it could be applied to any scored item type (e.g., short answer, script concordance) once a sufficiently large, human-calibrated bank becomes available.”

Classified as probably not a substantial claim.

The input format was standardized across all datasets to address prompt engineering influence.

“Recognizing the significant influence of prompt engineering on the outputs of generative models, we standardized the input format across all datasets.”

May combine two claims.

IRT places item parameters and examinee ability on a common interval scale, allowing direct comparison of abilities even when examinees respond to entirely different sets of items, forming the theoretical basis for CAT.

“IRT places item parameters and examinee ability on a common interval scale (typically Mean = 0, SD = 1). This allows for the direct comparison of abilities even when examinees respond to entirely different sets of items, forming the theoretical basis for Computerized Adaptive Testing28.”

• cites [28] van der Linden, W. J. & Glas, C. A. W. Capitalization on item calibration error in adaptive testing. Appl. Meas. Educ. 13, 35–53 (2000). sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Recent analyses have emphasized a persistent gap between strong benchmark results and the scarcity of rigorous clinical trials for medical AI, underscoring the risk of overreliance on offline evaluations when making deployment decisions.

“Recent analyses have emphasized the persistent gap between strong benchmark results and the scarcity of rigorous clinical trials for medical AI, underscoring the risk of overreliance on offline evaluations when making deployment decisions23.”

• cites [23] Bedi, S. et al. Testing and evaluation of health care applications of large language models. JAMA. 333, 319–328 (2025).. sourceNot checked — no full text of the source

May combine two claims.

CAT depends on IRT-derived item information and requires reliable item parameter estimates.

“CAT depends on IRT-derived item information and requires reliable item parameter estimates.”

May combine two claims.

The IRT assumptions support the validity of the calibrated item bank and scale for comparative benchmarking; they do not imply a human-like latent structure for LLMs.

“Accordingly, the IRT assumptions support the validity of the calibrated item bank and scale for comparative benchmarking; they do not imply a human-like latent structure for LLMs.”

• AssumptionRests on an assumption: LLMs are not human examinees and do not imply a single psychological latent trait.

Probably not a substantial claim: may not pass S3 (load-bearing).

The temperature parameter was fixed at 0.0 to ensure deterministic outputs.

“The temperature parameter was fixed at 0.0 to ensure deterministic outputs.”

May combine two claims.

Findings are anchored in a single nationally calibrated item bank, so generalizability may differ under other blueprints such as USMLE or European systems.

“our findings are anchored in a single nationally calibrated item bank, generalizability may differ under other blueprints such as the USMLE18 or European systems.”

• cites USMLE18Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: USMLE Step 1 emphasizes foundational biomedical mechanisms whereas Step 2 CK places greater weight on clinical diagnosis and management in vignette-based single best answer formats, which can shift item-parameter distributions and information functions.