Interrater reliability in the assessment of physiotherapy students
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Background: Reliable and objective assessment of psychomotor skills in physiotherapy students’ education is essential for direct feedback and skill improvement. The aim of this study is to determine the interrater reliability in the assessment process of physiotherapy students and to analyse the assessment behaviour of the examiners. Methods: Physiotherapy teachers from two different schools assessed students from two different schools performing proprioceptive neuromuscular facilitation (PNF) patterns. An evaluation sheet with a 6-point rating scale and 20 evaluation criteria including an overall rating was used for assessment. The interrater reliability was determined calculating an intraclass-correlation coefficient (ICC) and Krippendorff’s alpha. The assessment behaviour of the examiners was further analysed calculating the location parameters and showing the item response distribution over item in form of a Likert plot. Results: The ICC estimates were mostly below 0.4, indicating poor interrater reliability. This was confirmed by Krippendorff’s alpha. The examiners showed a certain central tendency and intergroup bias. Discussion and conclusion: The interrater reliability in this assessment format was rather low. No difference between the two physiotherapy schools concerning the interrater reliability could be identified. Despite certain limitations of this study, there is a definite need for improvement of the assessment process in physiotherapy education to provide the students with reliable and objective feedback and ensure a certain level of professional competence in the students. Trial registration: The study was approved by the ethics committee of the Medical Faculty RWTH Aachen University (EK 340/16).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Agreement was established according to the following inter‐rater reliability measures: a 70% consistency in scoring items as ‘important’ and ‘very important’; a mean value of 3.5; and an interquartile range ≤ 1 [31, 32, 33].”
From Co‐Designing a Multimodal Physical Activity Intervention for Individuals With Young‐Onset Type 2 Diabetes (18–40 Years) in China (2026) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Lee, K. M., Lee, J., Chung, C. Y., Ahn, S., Sung, K. H., Kim, T. W., Lee, H. J., & Park, M. S. (2012) · Clinics in Orthopedic Surgery · doi:10.4055/cios.2012.4.2.149 · Cited by 216 · cited 1×
“The ICC model we used was a one-way random effects, absolute agreement, single rater and multiple rater model, which usually produces lower ICC estimates than the consistency model mentioned above [21].”
Method
Research works printed but not linked: 34
- Wass V, der Vleuten CV, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357(9260):945–9. · DOI · Paper page
- Michels MEJ, Evans DE, Blok GA. What is a clinical skill? Searching for order in chaos through a modified Delphi process. Med Teach. 2012;34(8):e573–81. · DOI · Paper page
- Sattelmayer M, Elsig S, Hilfiker R, Baer G. A systematic review and meta-analysis of selected motor learning principles in physiotherapy and medical education. BMC Med Educ. 2016;16 Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4714441/ [cited 2019 Nov 20]. · DOI · Paper page
- Nicholls D, Sweet L, Muller A, Hyett J. Teaching psychomotor skills in the twenty-first century: revisiting and reviewing instructional approaches through the lens of contemporary literature. Med Teach. 2016;38(10):1056–63. · DOI · Paper page
- Larimore WL, Sapolsky BS. Maternity care in family medicine: economics and malpractice. J Fam Pract. 1995;40(2):153–60.
- Adler SS, Beckers D, Buck M. PNF in Practice. Berlin, Heidelberg: Springer Berlin Heidelberg; 2008. Available from: http://link.springer.com/10.1007/978-3-540-73904-3 [cited 2021 Oct 2] · DOI
- Hejri SM, Jalili M, Masoomi R, Shirazi M, Nedjat S, Norcini J. The utility of mini-clinical evaluation exercise in undergraduate and postgraduate medical education: a BEME review: BEME guide no. 59. Med Teach. 2020;42(2):125–42. · DOI
- Silva CCBM, Lunardi AC, Mendes FAR, Souza FFP, Carvalho CRF. Objective structured clinical evaluation as an assessment method for undergraduate chest physical therapy students: a cross-sectional study. Braz J Phys Ther. 2011;15(6):481–6. · DOI
- O’Connor A, McGarr O, Cantillon P, McCurtin A, Clifford A. Clinical performance assessment tools in physiotherapy practice education: a systematic review. Physiotherapy. 2018;104(1):46–53. · DOI
- Weigelt M, Güldenpenning I. In: Schüler J, Wegner M, Plessner H, editors. Lernen und Gedächtnis im Sport. Sportpsychologie: Springer-Verlag GmbH; 2020. p. 43–68.
- Westwater-Wood S, Adams N, Kerry R. The use of proprioceptive neuromuscular facilitation in physiotherapy practice. Phys Ther Rev. 2010;15:23–8. · doi:10.1179/174328810x12647087218677
- McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30–46. · DOI · Paper page
- Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–90. · DOI · Paper page
- Krippendorff K. Content analysis: an introduction to its methodology. Thousand Oaks: Sage Publications Inc; 2004. p. 446.
- Schuh LA, London Z, Neel R, Brock C, Kissela BM, Schultz L, et al. Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology. 2009;73(11):904–8. · DOI
- Hewstone M, Rubin M, Willis H. Intergroup Bias. Annu Rev Psychol. 2002;53(1):575–604. · DOI · Paper page
- Wirtz M, Caspar F. Beurteilerübereinstimmung und Beurteilerreliabilität. Göttingen: Hogrefe; 2002. p. 287.
- Müller R, Büttner P. A critical discussion of intraclass correlation coefficients. Stat Med. 1994;13(23–24):2465–76. · DOI · Paper page
- Wagner S, Helmreich I, Lieb K, Tadić A. Standardized rater training for the Hamilton depression rating scale (HAMD17) and the inventory of depressive symptoms (IDSC30). Psychopathology. 2011;44(1):68–70. · DOI
- Müller MJ, Rossbach W, Dannigkeit P, Müller-Siecheneder F, Szegedi A, Wetzel H. Evaluation of standardized rater training for the positive and negative syndrome scale (PANSS). Schizophr Res. 1998;32(3):151–60. · DOI
- Zhang EQ, Leung VS, Pang DS. Influence of rater training on inter- and Intrarater reliability when using the rat grimace scale. J Am Assoc Lab Anim Sci. 2019;58(2):178–83. · doi:10.30802/aalas-jaalas-18-000044
- Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of Mini-CEX scores: a randomized, controlled trial. J Gen Intern Med. 2009;24(1):74–9. · DOI · Paper page
- Robertson RL, Vergis A, Gillman LM, Park J. Effect of rater training on the reliability of technical skill assessments: a randomized controlled trial. Can J Surg. 2018;61(6):405–11. · doi:10.1503/cjs.015917
- Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J. 1975;1(5955):447–51. · DOI · Paper page
- Lee Y. OSCE for the medical licensing examination in Korea. Kaohsiung J Med Sci. 2008;24(12):646–50. · DOI
- Rushforth HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse Educ Today. 2007;27(5):481–90. · DOI · Paper page
- Faramarzi M, Salmalian H, Pasha H, Banihosseini Z. Validity and reliability of combination of observed practice with objective structured clinical examination (OSCE) in midwifery final exam. Annu Res Rev Biol. 2014;5(2):124–31. · doi:10.9734/arrb/2015/12775
- Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226–35. · DOI · Paper page
- Fleming E, Hole J, Patil V. Objective structured clinical examinations and reliability: are we there yet? Clin Teach. 2012;9(3):193–4. · doi:10.1111/j.1743-498x.2012.00574.x
- Judd BK, Scanlan JN, Alison JA, Waters D, Gordon CJ. The validity of a professional competence tool for physiotherapy students in simulation-based clinical education: a Rasch analysis. BMC Med Educ. 2016;16:196. · DOI
- Torres-Narváez M-R, Vargas-Pinilla O-C, Rodríguez-Grande E-I. Validity and reproducibility of a tool for assessing clinical competencies in physical therapy students. BMC Med Educ]. 2018;18. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6260878/ [cited 2019 Dec 13] · DOI
- Kutafina E, Laukamp D, Bettermann R, Schroeder U, Jonas SM. Wearable sensors for eLearning of manual tasks: using forearm EMG in hand hygiene training. Sensors. 2016;16(8):1221. · doi:10.3390/s16081221
- Rose M, Curtze C, O’Sullivan J, El-Gohary M, Crawford D, Friess D, et al. Wearable inertial sensors allow for quantitative assessment of shoulder and elbow kinematics in a cadaveric knee arthroscopy model. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2017;33(12):2110–6. · DOI
- Jovanovic M, Seiffarth J, Kutafina E, Jonas SM. IOS Press Ebooks - Automated Error Detection in Physiotherapy Training. Available from: http://ebooks.iospress.nl/publication/48978. [cited 2019 Mar 26]
Cited by
Zhao, X., Forbes, A., Ghazaleh, H. A., Cheng, L., Guo, X., & Duaso, M. (2026) · Health Expectations · doi:10.1111/hex.70580
Metadata
- Authors
- Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., Olmeo, A., Lucas, G., & Jonas, S. M.
- Venue
- BMC Medical Education
- Published
- 2022-03-16
- Volume
- 22
- Issue
- 1
- ISSN
- 1472-6920
- Cited by
- 7
- References
- 3
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- RWTH Aachen University
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interrater reliability in the assessment of physiotherapy students. https://app.cochat.ai/papers/10.1186/s12909-022-03231-y
- The paper
- Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., Olmeo, A., Lucas, G., & Jonas, S. M. (2022). Interrater reliability in the assessment of physiotherapy students. BMC Medical Education. https://doi.org/10.1186/s12909-022-03231-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.