The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Comparison between Inter-rater Reliability and Inter-rater Agreement in Performance Assessment
Open accessAbstract
Introduction: Over the years, performance assessment (PA) has been widely employed in medical education, Objective Structured Clinical Examination (OSCE) being an excellent example. Typically, performance assessment involves multiple raters, and therefore, consistency among the scores provided by the auditors is a precondition to ensure the accuracy of the assessment. Inter-rater agreement and inter-rater reliability are two indices that are used to ensure such scoring consistency. This research primarily examined the relationship between inter-rater agreement and inter-rater reliability. Materials and Methods: This study used 3 sets of simulated data that was based on raters’ evaluation of student performance to examine the relationship between inter-rater agreement and inter-rater reliability. Results: Data set 1 had high inter-rater agreement but low inter-rater reliability, data set 2 had high inter-rater reliability but low inter-rater agreement, and data set 3 had high inter-rater agreement and high inter-rater reliability. Conclusion: Inter-rater agreement and inter-rater reliability can but do not necessarily coexist. The presence of one does not guarantee that of the other. Inter-rater agreement and inter-rater reliability are both important for PA. The former shows stability of scores a student receives from different raters, while the latter shows the consistence of scores across different students from different raters. Key words: Consistency, Multiple-rater, Psychometrics
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter-rater agreement and inter-rater reliability are not interchangeable and represent different notions of reliability.
“However, these two terms are not interchangeable and represent different notions of reliability (e.g., Kottner et al., 2011;Liao, Hunt, Chen, 2010;Stolarova, Wolf, Rinker, Brielmann, 2014).”
From Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability (2018) · cites this paper as Liao, Hunt, Chen, 2010
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
rwg: an assessment of within-group interrater agreement
James, L. R., Demaree, R. G., & Wolf, G. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.2.306 · Cited by 2,221
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139
The use of scoring rubrics: Reliability, validity and educational consequences
Jonsson, A., & Svingby, G. (2007) · Educational Research Review · doi:10.1016/j.edurev.2007.05.002 · Cited by 1,509
Educational Assessment of Students
Nitko, A. J. (1996) · Cited by 1,033
Assessment of clinical competence
Wass, V., van der Vleuten, C. P. M., Shatzer, J. H., & Jones, R. F. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04221-5 · Cited by 1,023
Brook, R. H., McGlynn, E. A., & Cleary, P. D. (1996) · New England Journal of Medicine · doi:10.1056/nejm199609263351311 · Cited by 780
Authentic assessment of teaching in context
Darling-Hammond, L., & Snyder, J. (2000) · Teaching and Teacher Education · doi:10.1016/s0742-051x(00)00015-9 · Cited by 715
Detecting and measuring rater effects using many-facet Rasch measurement: Part II.
Myford, C. M., & Wolfe, E. W. (2003) · PubMed · Cited by 384
Rater bias in psychological research: When is it a problem and what can we do about it?
Hoyt, W. T. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.1.64 · Cited by 323
Assessing professionalism: a review of the literature
Lynch, D. C., Surdyk, P. M., & Eiser, A. R. (2004) · Medical Teacher · doi:10.1080/01421590410001696434 · Cited by 245
Objective structured clinical exams: a critical review.
Turner, J. L., & Dankoski, M. E. (2008) · PubMed · Cited by 237
Competency Based Education: A Bandwagon in Search of a Definition
Spady, W. G. (1977) · Educational Researcher · doi:10.3102/0013189x006001009 · Cited by 210
The Unintended Consequences of Measuring Quality on the Quality of Medical Care
Casalino, L. P. (1999) · New England Journal of Medicine · doi:10.1056/nejm199910073411511 · Cited by 200
Weigle, S. C. (1999) · Assessing Writing · doi:10.1016/s1075-2935(00)00010-6 · Cited by 157
The Role of Physicians in the Future of Quality Management
Blumenthal, D., & Epstein, A. M. (1996) · New England Journal of Medicine · doi:10.1056/nejm199610243351721 · Cited by 148
Challenges to the assessment of competence and competencies.
Lichtenberg, J. W., Portnoy, S. M., Bebeau, M. J., Leigh, I. W., Nelson, P. D., Rubin, N. J., Smith, I. L., & Kaslow, N. J. (2007) · Professional Psychology Research and Practice · doi:10.1037/0735-7028.38.5.474 · Cited by 143
Hutchinson, L., Aitken, P., & Hayes, T. M. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01120.x · Cited by 83
Performance Assessment and Electronic Portfolios: Their Effect on Teacher Learning and Education
Pecheone, R. L., Pigg, M., Wei, R. C., & Souviney, R. J. (2005) · The Clearing House A Journal of Educational Strategies Issues and Ideas · doi:10.3200/tchs.78.4.164-176 · Cited by 82
Setting performance standards for medical practice: a theoretical framework
Southgate, L. J., Hays, R., Norcini, J. J., Mulholland, H., Ayers, B., Woolliscroft, J., Cusimano, M. D., McAvoy, P. A., AINSWORTH, M. A., Haist, S. A., et al. (2001) · Medical Education · doi:10.1046/j.1365-2923.2001.00897.x · Cited by 63
Southgate, L. J., & Pringle, M. (1999) · BMJ · doi:10.1136/bmj.319.7218.1180 · Cited by 53
Southgate, L. J., & Dauphinee, D. W. (1998) · BMJ · doi:10.1136/bmj.316.7132.697 · Cited by 42
State Policy Related to Teacher Licensure
Youngs, P. A., Odden, A. R., & Porter, A. C. (2003) · Educational Policy · doi:10.1177/0895904803017002002 · Cited by 40
Clinical Performance Measurement—A Hard Sell
Jencks, S. F. (2000) · JAMA · doi:10.1001/jama.283.15.2015 · Cited by 38
Determining the Validity of Performance‐Based Assessment
Burger, S. E., & Burger, D. L. (1994) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1994.tb00779.x · Cited by 30
Psychometric Properties of Clinical Performance Ratings
Dielman, T. E., Hull, A. L., & Davis, W. K. (1980) · Evaluation & the Health Professions · doi:10.1177/016327878000300106 · Cited by 25
Assessment of Student Performance
Michael, J., & Modell, H. I. (2003) · Routledge eBooks · doi:10.4324/9781410609212-20 · Cited by 20
Jolly, B. C., McAvoy, P. A., & Southgate, L. J. (2001) · PubMed Central · Cited by 14
GMC's proposals for revalidation
Newble, D. I. (2001) · BMJ · doi:10.1136/bmj.322.7282.358 · Cited by 8
Flagged first, then by citations
Research works printed but not linked: 2
- Hopkins KD. Educational and Psychological Measurement and Evaluation, 8th ed. Needham Heights: Pearson, 1998.
- Black P. Testing: Friend or foe? London: Falmer Press, 1998.
Cited by
Sorted by citations
EURASIA Journal of Mathematics, Science and Technology Education
(2018) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.12973/eurasia · Cited by 300
Nakayama, M., Sears, C. R., Hino, Y., & Lupker, S. J. (2013) · Journal of Cognitive Psychology · doi:10.1080/20445911.2013.839560 · Cited by 100
Kumar, A. D., Kugler, J., & Jensen, T. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-04945-4 · Cited by 98
Stolarova, M., Wolf, C., Rinker, T., & Brielmann, A. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00509 · Cited by 93
Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations
Quah, B., Zheng, L., Sng, T. J. H., Yong, C. W., & Islam, I. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05881-6 · Cited by 87
Evidence-based answers to 15 questions about leveraging 360-degree feedback.
Nowack, K. M., & Mashihi, S. (2012) · Consulting psychology journal · doi:10.1037/a0030011 · Cited by 86
Insights into the Angoff method: results from a simulation study
Shulruf, B., WILKINSON, T. J., Weller, J. M., Jones, P., & Poole, P. (2016) · BMC Medical Education · doi:10.1186/s12909-016-0656-7 · Cited by 52
Shattuck, K., Zimmerman, W. A., & Adair, D. (2014) · Internet Learning · doi:10.18278/il.3.1.3 · Cited by 39
Perceptions of and Behavior toward University Students with Autism
Lipson, J., Taylor, C., Burk, J. A., & Dickter, C. L. (2020) · Basic and Applied Social Psychology · doi:10.1080/01973533.2020.1785468 · Cited by 30
Yimer, S. T., & Feza, N. N. (2019) · International Electronic Journal of Mathematics Education · doi:10.29333/iejme/5936 · Cited by 27
Point-of-Care-ultrasound in undergraduate medical education: a scoping review of assessment methods
DeBiasio, C., Pageau, P., Shefrin, A. E., Woo, M. Y., & Cheung, W. J. (2023) · The Ultrasound Journal · doi:10.1186/s13089-023-00325-6 · Cited by 26
Exploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability
Wilhelm, A. G., Rouse, A. G., & Jones, F. (2018) · University of Massachusetts Amherst · doi:10.7275/at67-md25 · Cited by 20
Daly, M., Salamonson, Y., Glew, P. J., & Everett, B. (2016) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2016.09.009 · Cited by 22
In the minds of OSCE examiners: uncovering hidden assumptions
Chahine, S., Holmes, B., & Kowalewski, Z. (2015) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9655-4 · Cited by 21
Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance
Gontar, P., & Hoermann, H.-J. (2015) · International Journal of Aviation Psychology · doi:10.1080/10508414.2015.1162636 · Cited by 21
Bajaj, J. S., Frederick, R. T., Bass, N. M., Ghabril, M., Coyne, K. S., Margolis, M. K., Santoro, M., Coakley, D. F., Mokhtarani, M., Jurek, M., et al. (2016) · Metabolic Brain Disease · doi:10.1007/s11011-016-9851-9 · Cited by 21
Yimer, S. T. (2020) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/8705 · Cited by 18
Duran, C. A., Shames, M. L., Bismuth, J. X., & Lee, J. T. (2013) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2013.07.005 · Cited by 18
Kuzminsky, J., Phillips, H., Sharif, H., Moran, C. A., Gleason, H. E., Topulos, S. P., Pitt, K., McNeil, L. K., McCoy, A. M., & Kesavadas, T. (2023) · American Journal of Veterinary Research · doi:10.2460/ajvr.23.03.0058 · Cited by 17
Delvaux, V., & Pillot-Loiseau, C. (2019) · Journal of Voice · doi:10.1016/j.jvoice.2019.02.013 · Cited by 16
Liu, H. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14961 · Cited by 15
An Operative Performance Rating System for Urology Residents
Benson, A. D., Markwell, S. J., Köhler, T., & Tarter, T. H. (2012) · The Journal of Urology · doi:10.1016/j.juro.2012.07.047 · Cited by 14
Dereje, E. W. (2023) · Journal of Pedagogical Sociology and Psychology · doi:10.33902/jpsp.202321592 · Cited by 13
On the Consistency of Automatic Scoring with Large Language Models
Xue, M., Xiao, X., Liu, Y., & Wilson, M. (2026) · Educational and Psychological Measurement · doi:10.1177/00131644261418138 · Cited by 10
Assessing Operative Skill in the Competency-based Education Era
Toale, C., Morris, M., & Kavanagh, D. O. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000005242 · Cited by 10
Lin, C., Chang, J. Z., Hsu, T.-C., Liu, Y.-J., Yu, S.-H., Tsai, S.-L., Lai, E. H., & Lin, C. (2013) · Journal of Dental Sciences · doi:10.1016/j.jds.2013.01.002 · Cited by 10
Retention in MOOCS: some key factors
Chiappe, A., & Castillo, B. D. L. (2020) · Ensaio Avaliação e Políticas Públicas em Educação · doi:10.1590/s0104-40362020002802667 · Cited by 9
Fillers as Communication Strategies Among English Second Language Speakers in Job Interviews
Hassan, W. N. F. W., Awang, S. B., & Abdullah, N. B. (2021) · International Journal of English Linguistics · doi:10.5539/ijel.v12n1p30 · Cited by 9
TractEM: Evaluation of protocols for deterministic tractography white matter atlas
Rheault, F., Bayrak, R. G., Wang, X., Schilling, K. G., Greer, J. M., Hansen, C. B., Kerley, C. I., Ramadass, K., Remedios, L. W., Blaber, J. A., et al. (2021) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2021.10.017 · Cited by 7
Mediating effect of cognitive load in song learning with visually presented lyrics
Han, Y.-J. (2020) · Psychology of Music · doi:10.1177/0305735620959430 · Cited by 6
Dereje, E. W., Ayele, M. A., & Woldemichael, M. (2023) · Pedagogical Research · doi:10.29333/pr/13332 · Cited by 5
Use of Fillers as a Communication Strategy in Oral Interactions
Awang, S. B., Hassan, W. N. F. W., Abdullah, N. B., & Zawawi, M. Z. M. (2022) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v12-i7/14330 · Cited by 5
Gruber, Á., Cserkész, T., Kiss, C., & Lanszki, J. (2024) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2024.106472 · Cited by 5
Watari, T., Koyama, S., Kato, Y., Paku, Y., Kanada, Y., & Sakurai, H. (2022) · PubMed Central · doi:10.20407/fmj.2021-010 · Cited by 5
Inter-rater and Intra-Rater Reliability Test with Goodenough-Harris Drawing Test
Tarigan, M., & Fadillah, F. (2022) · The Open Psychology Journal · doi:10.2174/18743501-v15-e2207130 · Cited by 5
Inter-rater Agreement of End-of-shift Evaluations Based on a Single Encounter
Warrington, S., Beeson, M. S., & Bradford, A. (2017) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2016.12.32014 · Cited by 4
Evaluation of Usability Problems of Labor Portal in Saudi ArabiaPreprint
Al-Zuabi, H., & Alshaikhli, I. F. T. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2372783 · Cited by 4
Alves, N. D. C., von Wangenheim, C. G., Martins-Pacheco, L. H., & Borgatto, A. F. (2021) · doi:10.5753/educomp.2021.14467 · Cited by 3
Merino‐Soto, C., la Cruz, G. C.-D., & Medina, E. (2016) · Revista Latinoamericana de Psicología · doi:10.1016/j.rlp.2015.09.011 · Cited by 3
Motivational Interviewing Training: A Pilot Study in a Master's Level Counseling Program
Zeligman, M. R., Dispenza, F., Chang, C. Y., Levy, D. B., McDonald, C. P., & Murphy, T. M. (2017) · Counseling Outcome Research and Evaluation · doi:10.1080/21501378.2017.1342518 · Cited by 3
A framework for land information management in Ghana
Adiaba, S. Y. (2014) · Wolverhampton Intellectual Repository and E-Theses (University of Wolverhampton) · Cited by 3
VASCULAR & INTERVENTIONAL RADIOLOGY
Noaparast, M., Rabani, A., Karimian, F., Jafarian, A., Vaezi, F., & Ghanaati, H. (2011) · Cited by 3
Chu, S., Lan, Y., Kao, S.-P., Chen, T., & Hsieh, M.-C. (2012) · Tzu Chi Medical Journal · doi:10.1016/j.tcmj.2012.02.003 · Cited by 2
Liao, S.-C., Hung, Y., Chang, C.-R., & Ting, Y.-X. (2025) · Advances in Medical Education and Practice · doi:10.2147/amep.s524693 · Cited by 2
Self-citationNoaparast, M., Rabani, A., Karimian, F., Bodaghabadi, M., Aran, S., Mirsharifi, R., Jafarian, A., Vaezi, F., & Ghanaati, H. (2011) · Europe PMC (PubMed Central) · Cited by 2
Orlando, J., Joshi, A. R. T., Carter, M., May, H., Bortle, C., & White, P. (2016) · International Journal of Academic Medicine · doi:10.4103/2455-5568.196870 · Cited by 2
Awang, S. B., Hassan, W. N. F. W., Abdullah, N. B., Zakaria, W. N. F. W., & Razak, S. S. (2022) · English Language Teaching · doi:10.5539/elt.v15n10p64 · Cited by 2
Chang, Y., Campbell, M., Yen, W., Yang, C., Peng, C., & Chen, Y. (2020) · Heart & Lung · doi:10.1016/j.hrtlng.2020.11.005 · Cited by 1
Pongrácz, P., Lugosi, C. A., Kovács, P. A., & Lanszki, J. (2025) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2025.106806 · Cited by 1
Reliability in Industry-Academia Assessment: A Framework for Validating Competencies in IT Education
Jakkaew, P., Putjorn, P., & Chondamrongkul, N. (2026) · Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON) · doi:10.1109/ectidamtncon67592.2026.11460083 · Cited by 1
Reportovanie súhlasu posudzovateľov a spoľahlivosti posudzovateľov
Kočišová, L. (2022) · TESTFÓRUM · doi:10.5817/tf2022-15-14647 · Cited by 1
Jaiyote, S. (2016) · University of Bedfordshire Repository (University of Bedfordshire) · Cited by 1
An Interesting Case Of Concomitant Tuberculosis and Adenocarcinoma Colon
Monteiro, R., Agrawal, P. B., Sujanani, S., & Vyas, K. C. (2020) · Annals of Pathology and Laboratory Medicine · doi:10.21276/apalm.2667
A Usability Model for Software Development Processes and Practices
Fontdevila, D. (2021) · doi:10.35537/10915/121169
Goldstein, L. J. K. (2019) · YorkSpace (York University)
Lee, H.-Y., Yune, S.-J., Lee, S. Y., & Im, S. J. (2021) · Korean Medical Education Review · doi:10.17496/kmer.2021.23.3.185
The cognitive structure of gross anatomy knowledge in physiotherapy students
Besselink, A. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2519
DeBiasio, C., Pageau, P., Shefrin, A. E., Woo, M. Y., & Cheung, W. J. (2024) · Figshare · doi:10.6084/m9.figshare.26596376
DeBiasio, C., Pageau, P., Shefrin, A. E., Woo, M. Y., & Cheung, W. J. (2024) · Figshare · doi:10.6084/m9.figshare.26596379
Fontdevila, D., Genero, M., Paéz, N., & Oliveros, A. (2023) · Research Square · doi:10.21203/rs.3.rs-3104393/v1
Evaluating artificial intelligence-generated multiple-choice questions in clinical pathology
Ismail, A. A. O., Ma, W., Dunbar, N. M., Geno, K. A., Karrs, J. X., & Martin, I. W. (2026) · Academic Pathology · doi:10.1016/j.acpath.2026.100275
Yang, J., Zhang, Y., Wang, Y., Deng, Y., & Zhang, Y. (2026) · SAGE Open · doi:10.1177/21582440261466029
Ardakani, B. Y., Dehghani-Ahmadabad, A., Zareipour, M., Mazloomi-Mahmoodabad, S. S., Nabimeybodi, R., Ardakani, M. F., & Vakili, M. (2026) · Critical Public Health · doi:10.1080/09581596.2026.2727197
Dam, D. J. V., Gidaro, R., Ferreira, S., Morogiello, J. M., Jaffe, D., & Semmel, A. D. (2026) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000005673
Metadata
- Authors
- Liao, S. C., Hunt, E. A., & Chen, W.
- Affiliations
- China Medical University (TW)
- Johns Hopkins University (US)
- Venue
- Annals of the Academy of Medicine, Singapore
- Publisher
- Academy of Medicine, Singapore
- Published
- 2010-08-15
- Volume
- 39
- Issue
- 8
- Pages
- 613-618
- ISSN
- 0304-4602 (ISSN-L)
- OpenAlex
- W118576434
- Cited by
- 65
- References
- 29
- Citations by year
- 2026: 6 · 2025: 2 · 2024: 5 · 2023: 5 · 2022: 5 · 2021: 7 · 2020: 7 · 2019: 4 · 2018: 2 · 2017: 2 · 2016: 6 · 2015: 2 · 2014: 3 · 2013: 4 · 2012: 3
- FWCI
- 3.00
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Annals of the Academy of Medicine Singapore (publishedVersion)
- PubMed (publishedVersion)
- Funders
- China Medical University Hospital (CMUH-EDU97-11E)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Medical Education and Admissions
- Keywords
- inter-rater reliability, interobserver agreement, performance evaluation (human resources), OSCE, rater consistency, score stability, medical education assessment, simulated data
- MeSH terms
- Clinical Competence, Curriculum, Education, Medical, Humans, Psychometrics, Statistics as Topic, Students, Medical, Task Performance and Analysis, Reproducibility of Results, Statistics, Nonparametric, Validation Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
