Anchor-Based Methods for Judgmentally Estimating Item Statistics
Abstract
One important lesson learned from the controversy several years ago between the Educational Testing Service (ETS) and Stanley H. Kaplan Educational Centers Ltd. about the computerbased version of the Graduate Record Examination (GRE) was that larger item banks than had been initially assumed necessary are needed to support flexibly scheduled computer-based tests. Large item banks are needed to keep conditional item exposure levels low to protect the validity of test scores (Davey & Nering, 2002). Of course, one consequence is the increased cost to produce the extra test items, and another is the increased cost to field test substantially more test items.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These discrepancies between the absolute difficulty index and rating could be due to differences in experts' opinions about items, experiences, and the process of estimating the difficulty of test items [59].
“These discrepancies between the absolute difficulty index and rating could be due to differences in experts’ opinions about items, experiences, and the process of estimating the difficulty of test items [59].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [59]
There are variations and contradictions in experts' judgments regarding item difficulty in the literature [58-60].
“There are variations and contradictions in experts’ judgments regarding item difficulty in the literature [58–60].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [59]
Cites
P1: FBQ/JZK P2:
(n.d.)
Unresolved
Flagged first, then by citations
1 of the 1 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
AI in medical education: uses of AI in construction type A MCQs
Rezigalla, A. A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05250-3 · Cited by 36 · see citing works
The difference between estimated and perceived item difficulty: An empirical study
SAYIN, A., & Bulut, O. (2024) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.1376160 · Cited by 4 · see citing works
Kõlamets, L., Kasuk, H., Holbrook, J. B., et al. (2026) · Chemistry Teacher International · doi:10.1515/cti-2025-0038
- Authors
- Hambleton, R. K., & Jirka, S. J.
- Venue
- Handbook of Test Development
- Publisher
- Routledge
- Published
- 2011-04-26
- Pages
- 413-434
- OpenAlex
- W4230876005
- Cited by
- 4
- References
- 1
- Citations by year
- 2026: 1 · 2024: 2 · 2019: 1
- FWCI
- 0.00
- Type
- book chapter
- Language
- en
- Open access
- closed
- Locations
- Routledge eBooks (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Assessment and Pedagogy
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- anchor-based methods, computer-based testing, item bank, test score validity, item response theory, exam question design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
