Angoff′s method: The impact of raters′ selection
Open accessAbstract
Background: Several methods have been proposed for setting an examination pass mark (PM), and the Angoff's method or its modified version is the preferred one. Selection of raters is important and affects the PM. Aims and Objectives: This study aims to investigate the selection of raters in the Angoff's method and the impact of academic degrees and experience on the PM decided on. Materials and Methods: Type A MCQs examination was used in this study as a model. Raters with different academic degrees and experience participated in the study. Raters estimations were statiscally analyzed. Results: The selection of raters was crucial. Agreement among raters could be achieved by those with relevant qualifications and expertise. There was an association between high estimation, academic degree, expertise and high PM. Conclusion: Selection of raters for the Angoff's method should include those with different academic degrees, backgrounds and experience so that a satisfactory PM may be reached by means of a reasonable agreement.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter-rater agreement for expert rating was limited to 30%, and deviating experts were asked to rerate.
“Inter-rater agreement for the expert rating was limited to 30%. In cases of deviation in ratings, the concerned expert is requested to rerate according to other expert ratings or keep his rates [33].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [33]
The rating of item difficulty was assessed based on modified Angoff methods.
“The rating of item difficulty was assessed based on modified Angoff methods (Table 1) [33, 34].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [33]
Cited by
Sorted by citations
AI in medical education: uses of AI in construction type A MCQs
Rezigalla, A. A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05250-3 · Cited by 36 · see citing works
“In cases of deviation in ratings, the concerned expert is requested to rerate according to other expert ratings or keep his rates [33].”
BackgroundMethodSelf-citationBurrell, S. A., Ross, J. G., D’Annunzio, C., et al. (2021) · Journal of Cancer Education · doi:10.1007/s13187-021-02096-x · Cited by 20 · see citing works
Burrell, S. A., Ross, J. G., Byrne, C., et al. (2022) · Journal of Cancer Education · doi:10.1007/s13187-022-02199-z · Cited by 13 · see citing works
Rezigalla, A. A., Eleragi, A. M. S., Adam, M. I. E., & Mohammed, A. (2020) · Sudan Journal of Medical Sciences · doi:10.18502/sjms.v15i2.5503 · Cited by 4 · see citing works
Self-citationHashim, R. A., Shanmugam, S. K. S., Zubairi, A. M., et al. (2022) · Cogent Education · doi:10.1080/2331186x.2022.2104011 · Cited by 3 · see citing works
Introducing a Comprehensive Framework for Competency-based Procedure Training
Meliagros, P., & Forrest, R. (2025) · Journal of General Internal Medicine · doi:10.1007/s11606-025-09677-2 · Cited by 2 · see citing works
“25 Residents must achieve these standards before pursuing key components of procedures in the clinical environment.”
Ho, T. K., Kassim, N. L. A., O'Malley, L. A., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07822-3 · Cited by 1 · see citing works
“Previous research has shown that judges with higher academic qualifications and expertise are associated with greater variability and higher passing scores [83, 85].”
Standard Setting in Dental Education Assessment
Ho, T. K., Roudsari, R. V., & Kassim, N. L. A. (2025) · Springer eBooks · doi:10.1007/978-981-97-0275-6_44-1
Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus GroupsPreprint
Ho, T. K., O'Malley, L. A., & Roudsari, R. V. (2025) · Research Square · doi:10.21203/rs.3.rs-7347939/v1
Standard Setting in Dental Education Assessment
Ho, T. K., Roudsari, R. V., & Kassim, N. L. A. (2025) · Springer eBooks · doi:10.1007/978-981-97-0275-6_44-2
Refining competency benchmarks: a scoping review of Angoff standard-setting in dental education
Lin, G. S. S., Hisham, A. R. B., Majid, M. N. A., et al. (2026) · BMC Oral Health · doi:10.1186/s12903-026-08195-9
“The traditional Angoff method, which involves panels estimating the probability that a minimally competent candidate would answer each test item correctly, has been widely accepted for its structured approach [37, 38].”
- Authors
- Rezigalla, A.
- Venue
- Saudi Journal of Medicine and Medical Sciences
- Published
- 2015
- Volume
- 3
- Issue
- 3
- Pages
- 220
- ISSN
- 1658-631X (ISSN-L)
- OpenAlex
- W3092117207
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 1 · 2022: 2 · 2021: 1 · 2020: 1
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by-nc-sa
- Locations
- Saudi Journal of Medicine and Medical Sciences (publishedVersion, cc-by-nc-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Assessment and Pedagogy
- Keywords
- Angoff method, rater agreement, academic degrees, rater experience, multiple-choice questions, rater expertise
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
