The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
Angoff methods in standard setting in health professional education: a systematic review and meta-analysis
Abstract
BACKGROUND: Standard setting is a critical component of assessment in health professions education, ensuring fairness, defensibility, and validity in determining minimum competence. Among criterion-referenced approaches, the Angoff method and its variants are widely applied. However, the evidence base on their comparative performance remains fragmented, with no recent quantitative synthesis available. This systematic review and meta-analysis aimed to evaluate the outcomes of Angoff methods across different educational and assessment contexts. METHODS: A systematic electronic search was conducted for studies that applied Angoff or its variants [Modified, Yes/No, Three-level, Group (Yes/No), or Mastery] in health professions assessments and reported at least one of the following: cut score, pass rate, or inter-rater reliability. Pooled estimates were calculated using generalized linear mixed models and random-effects meta-analysis. Odds ratio (OR) was used as the effect estimate for pass rate in comparative studies. Meta-regression analyses explored heterogeneity by number of judges, and test length. RESULTS: A total of 91 studies were included in the systematic review. Mixed treatment comparison pooled estimates revealed significantly higher pass rates with Angoff (OR: 7.48) compared to the conventional fixed method. Mastery Angoff (88.2%) and Mastery Angoff with reality check (86.9%) were associated with higher cut scores, resulting in substantially lower pass rates (61.44% for Mastery Angoff). The Modified Angoff with reality check achieved excellent reliability (r = 0.917) while the Angoff (Yes/No) yielded the weakest and most variable results (r = 0.536), a finding confirmed by bootstrap analyses. Meta-regression indicated that each additional judge was associated with a 0.19-percentage point increase in cut scores (p = 0.003) while each additional test item was associated with a 0.05-percentage point increase in the pass rate (p = 0.001). CONCLUSION: For educators and policymakers, these findings underscore that the Angoff method is not a single technique but a flexible family of approaches with distinct outcomes. The choice of variants allows for the deliberate setting of more lenient or stringent standards, with Modified Angoff incorporating a reality check offering the most reliable and defensible results. This flexibility is a key strength but also a critical limitation, as an uninformed choice can lead to unintended pass rates. Therefore, selecting a specific Angoff variant must be a conscious decision aligned with the assessment's purpose and context.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Validating the interpretations and uses of test scores
Kane, M. T. (2013) · Journal of Educational Measurement · doi:10.1111/jedm.12000 · Cited by 2,110
Sterne, J. A. C., & Harbord, R. M. (2004) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0400400204 · Cited by 708 · cited 1×
“Publication bias could not be assessed as none of the comparisons had at least ten studies [13].”
GRADE approach to rate the certainty from a network meta-analysis: addressing incoherence
Brignardello-Petersen, R., Mustafa, R. A., Siemieniuk, R. A. C., Murad, M. H., Agoritsas, T., Izcovich, A., Schünemann, H. J., & Guyatt, G. H. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.11.025 · Cited by 202
Dwyer, T., Wright, S., Kulasegaram, K. M., Theodoropoulos, J., Chahal, J., Wasserstein, D., Ringsted, C., Hodges, B., & Ogilvie-Harris, D. (2016) · BMC Medical Education · doi:10.1186/s12909-015-0506-z · Cited by 201
Network meta-analysis explained
Dias, S., & Caldwell, D. M. (2019) · Archives of Disease in Childhood - Fetal and Neonatal Edition · doi:10.1136/archdischild-2018-315224 · Cited by 172
Do students have sufficient knowledge of clinical anatomy?
Prince, K. J. A. H., Scherpbier, A. J. J. A., van Mameren, H., Drukker, J., & van der Vleuten, C. P. M. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02096.x · Cited by 152
A Comparison of Standard-setting Procedures for an OSCE in Undergraduate Medical Education
Kaufman, D. M., Mann, K. V., Muijtjens, A. M. M., & van der Vleuten, C. P. M. (2000) · Academic Medicine · doi:10.1097/00001888-200003000-00018 · Cited by 144
Setting Cut-Scores: A Critical Review of the Angoff and Modified Angoff Methods
Ricker, K. L. (2006) · Alberta Journal of Educational Research · doi:10.55016/ojs/ajer.v52i1.55111 · Cited by 118
Comparison of a rational and an empirical standard setting procedure for an OSCE
Kramer, A. W. M., Muijtjens, A. M. M., Jansen, K. J., Düsman, H., Tan, L., & van der Vleuten, C. P. M. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01429.x · Cited by 116
Research works printed but not linked: 11
- Mortaz Hejri S, Jalili M. Standard setting in medical education: fundamental concepts and emerging challenges. Med J Islam Repub Iran. 2014;28:34.
- Park J. Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean medical licensing examination: a simulation study. J Educ Eval Health Prof. 2022;19:23. · DOI
- Anderson HG Jr, Nelson AA. Reliability and credibility of progress test criteria developed by alumni, faculty, and mixed alumni-faculty judge panels. Am J Pharm Educ. 2011;75(10):200. · DOI
- Chan SC, Amin SM, Lee TW. Implementing standard setting into the conjoint MAFP/FRACGP part 1 examination–process and issues. Malaysian Family Physician: Official J Acad Family Physicians Malaysia. 2016;11(2–3):2.
- Kamath MG, Pallath V, Ramnarayan K, Kamath A, Torke S, Gonsalves J. Standard setting of objective structured practical examination by modified Angoff method: A pilot study. Natl Med J India. 2016;29(3):160.
- Lee M, Hernandez E, Brook R, Ha E, Harris C, Plesa M, et al. Competency-based standard setting for a high-stakes objective structured clinical examination (OSCE): validity evidence. MedEdPublish. 2018;7:200. · DOI
- Park J, Yim MK, Kim NJ, Ahn DS, Kim YM. Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean medical licensing examination. J Educ Eval Health Prof. 2020;17:28. · doi:10.3352/jeehp.2020.17.28
- Park J. Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean medical licensing examination: a simulation study. J Educ Eval Health Prof. 2022;19:23. · DOI
- Sam AH, Millar KR, Westacott R, Melville CR, Brown CA. Standard setting very short answer questions (VSAQs) relative to single best answer questions (SBAQs): does having access to the answers make a difference? BMC Med Educ. 2022;22(1):640. · doi:10.1186/s12909-022-03693-0
- Tavakol M, O’Brien D, Stewart C. Determining intra-standard-setter inconsistency in the Angoff method using the three-parameter item response theory. Int J Med Educ. 2023;14:123. · doi:10.5116/ijme.64ed.e296
- Yim MK, Shin S. Using the Angoff method to set a standard on mock exams for the Korean nursing licensing examination. J Educ Eval Health Prof. 2020;17:14. · doi:10.3352/jeehp.2020.17.14
Metadata
- Authors
- Sridharan, K., & Sivaramakrishnan, G.
- Venue
- BMC Medical Education
- Published
- 2025-12-29
- Volume
- 25
- Issue
- 1
- ISSN
- 1472-6920
- Cited by
- 9
- References
- 99
- Language
- en
- Licence
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Angoff methods in standard setting in health professional education: a systematic review and meta-analysis. https://app.cochat.ai/papers/10.1186/s12909-025-08300-6
- The paper
- Sridharan, K., & Sivaramakrishnan, G. (2025). Angoff methods in standard setting in health professional education: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-025-08300-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.