CoChat

Interrater reliability in the assessment of physiotherapy students

Gittinger, Flora P., Lemos, Martin, Neumann, Jan L., Förster, Jürgen, Dohmen, Daniel, Berke, Birgit, Olmeo, Anke, Lucas, Gisela, Jonas, Stephan M. · 2022 · BMC Medical Education
Cited by 7

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Abstract

Background: Reliable and objective assessment of psychomotor skills in physiotherapy students’ education is essential for direct feedback and skill improvement. The aim of this study is to determine the interrater reliability in the assessment process of physiotherapy students and to analyse the assessment behaviour of the examiners. Methods: Physiotherapy teachers from two different schools assessed students from two different schools performing proprioceptive neuromuscular facilitation (PNF) patterns. An evaluation sheet with a 6-point rating scale and 20 evaluation criteria including an overall rating was used for assessment. The interrater reliability was determined calculating an intraclass-correlation coefficient (ICC) and Krippendorff’s alpha. The assessment behaviour of the examiners was further analysed calculating the location parameters and showing the item response distribution over item in form of a Likert plot. Results: The ICC estimates were mostly below 0.4, indicating poor interrater reliability. This was confirmed by Krippendorff’s alpha. The examiners showed a certain central tendency and intergroup bias. Discussion and conclusion: The interrater reliability in this assessment format was rather low. No difference between the two physiotherapy schools concerning the interrater reliability could be identified. Despite certain limitations of this study, there is a definite need for improvement of the assessment process in physiotherapy education to provide the students with reliable and objective feedback and ensure a certain level of professional competence in the students. Trial registration: The study was approved by the ethics committee of the Medical Faculty RWTH Aachen University (EK 340/16).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12909-022-03231-y

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 34

  1. Wass V, der Vleuten CV, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357(9260):945–9. · DOI · Paper page
  2. Michels MEJ, Evans DE, Blok GA. What is a clinical skill? Searching for order in chaos through a modified Delphi process. Med Teach. 2012;34(8):e573–81. · DOI · Paper page
  3. Sattelmayer M, Elsig S, Hilfiker R, Baer G. A systematic review and meta-analysis of selected motor learning principles in physiotherapy and medical education. BMC Med Educ. 2016;16 Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4714441/ [cited 2019 Nov 20]. · DOI · Paper page
  4. Nicholls D, Sweet L, Muller A, Hyett J. Teaching psychomotor skills in the twenty-first century: revisiting and reviewing instructional approaches through the lens of contemporary literature. Med Teach. 2016;38(10):1056–63. · DOI · Paper page
  5. Larimore WL, Sapolsky BS. Maternity care in family medicine: economics and malpractice. J Fam Pract. 1995;40(2):153–60.
  6. Adler SS, Beckers D, Buck M. PNF in Practice. Berlin, Heidelberg: Springer Berlin Heidelberg; 2008. Available from: http://link.springer.com/10.1007/978-3-540-73904-3 [cited 2021 Oct 2] · DOI
  7. Hejri SM, Jalili M, Masoomi R, Shirazi M, Nedjat S, Norcini J. The utility of mini-clinical evaluation exercise in undergraduate and postgraduate medical education: a BEME review: BEME guide no. 59. Med Teach. 2020;42(2):125–42. · DOI
  8. Silva CCBM, Lunardi AC, Mendes FAR, Souza FFP, Carvalho CRF. Objective structured clinical evaluation as an assessment method for undergraduate chest physical therapy students: a cross-sectional study. Braz J Phys Ther. 2011;15(6):481–6. · DOI
  9. O’Connor A, McGarr O, Cantillon P, McCurtin A, Clifford A. Clinical performance assessment tools in physiotherapy practice education: a systematic review. Physiotherapy. 2018;104(1):46–53. · DOI
  10. Weigelt M, Güldenpenning I. In: Schüler J, Wegner M, Plessner H, editors. Lernen und Gedächtnis im Sport. Sportpsychologie: Springer-Verlag GmbH; 2020. p. 43–68.
  11. Westwater-Wood S, Adams N, Kerry R. The use of proprioceptive neuromuscular facilitation in physiotherapy practice. Phys Ther Rev. 2010;15:23–8. · doi:10.1179/174328810x12647087218677
  12. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30–46. · DOI · Paper page
  13. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–90. · DOI · Paper page
  14. Krippendorff K. Content analysis: an introduction to its methodology. Thousand Oaks: Sage Publications Inc; 2004. p. 446.
  15. Schuh LA, London Z, Neel R, Brock C, Kissela BM, Schultz L, et al. Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology. 2009;73(11):904–8. · DOI
  16. Hewstone M, Rubin M, Willis H. Intergroup Bias. Annu Rev Psychol. 2002;53(1):575–604. · DOI · Paper page
  17. Wirtz M, Caspar F. Beurteilerübereinstimmung und Beurteilerreliabilität. Göttingen: Hogrefe; 2002. p. 287.
  18. Müller R, Büttner P. A critical discussion of intraclass correlation coefficients. Stat Med. 1994;13(23–24):2465–76. · DOI · Paper page
  19. Wagner S, Helmreich I, Lieb K, Tadić A. Standardized rater training for the Hamilton depression rating scale (HAMD17) and the inventory of depressive symptoms (IDSC30). Psychopathology. 2011;44(1):68–70. · DOI
  20. Müller MJ, Rossbach W, Dannigkeit P, Müller-Siecheneder F, Szegedi A, Wetzel H. Evaluation of standardized rater training for the positive and negative syndrome scale (PANSS). Schizophr Res. 1998;32(3):151–60. · DOI
  21. Zhang EQ, Leung VS, Pang DS. Influence of rater training on inter- and Intrarater reliability when using the rat grimace scale. J Am Assoc Lab Anim Sci. 2019;58(2):178–83. · doi:10.30802/aalas-jaalas-18-000044
  22. Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of Mini-CEX scores: a randomized, controlled trial. J Gen Intern Med. 2009;24(1):74–9. · DOI · Paper page
  23. Robertson RL, Vergis A, Gillman LM, Park J. Effect of rater training on the reliability of technical skill assessments: a randomized controlled trial. Can J Surg. 2018;61(6):405–11. · doi:10.1503/cjs.015917
  24. Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J. 1975;1(5955):447–51. · DOI · Paper page
  25. Lee Y. OSCE for the medical licensing examination in Korea. Kaohsiung J Med Sci. 2008;24(12):646–50. · DOI
  26. Rushforth HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse Educ Today. 2007;27(5):481–90. · DOI · Paper page
  27. Faramarzi M, Salmalian H, Pasha H, Banihosseini Z. Validity and reliability of combination of observed practice with objective structured clinical examination (OSCE) in midwifery final exam. Annu Res Rev Biol. 2014;5(2):124–31. · doi:10.9734/arrb/2015/12775
  28. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226–35. · DOI · Paper page
  29. Fleming E, Hole J, Patil V. Objective structured clinical examinations and reliability: are we there yet? Clin Teach. 2012;9(3):193–4. · doi:10.1111/j.1743-498x.2012.00574.x
  30. Judd BK, Scanlan JN, Alison JA, Waters D, Gordon CJ. The validity of a professional competence tool for physiotherapy students in simulation-based clinical education: a Rasch analysis. BMC Med Educ. 2016;16:196. · DOI
  31. Torres-Narváez M-R, Vargas-Pinilla O-C, Rodríguez-Grande E-I. Validity and reproducibility of a tool for assessing clinical competencies in physical therapy students. BMC Med Educ]. 2018;18. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6260878/ [cited 2019 Dec 13] · DOI
  32. Kutafina E, Laukamp D, Bettermann R, Schroeder U, Jonas SM. Wearable sensors for eLearning of manual tasks: using forearm EMG in hand hygiene training. Sensors. 2016;16(8):1221. · doi:10.3390/s16081221
  33. Rose M, Curtze C, O’Sullivan J, El-Gohary M, Crawford D, Friess D, et al. Wearable inertial sensors allow for quantitative assessment of shoulder and elbow kinematics in a cadaveric knee arthroscopy model. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2017;33(12):2110–6. · DOI
  34. Jovanovic M, Seiffarth J, Kutafina E, Jonas SM. IOS Press Ebooks - Automated Error Detection in Physiotherapy Training. Available from: http://ebooks.iospress.nl/publication/48978. [cited 2019 Mar 26]

Cited by

Metadata

Authors
Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., Olmeo, A., Lucas, G., & Jonas, S. M.
Venue
BMC Medical Education
Published
2022-03-16
Volume
22
Issue
1
ISSN
1472-6920
Cited by
7
References
3
Language
en
Licence
Funders
RWTH Aachen University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Interrater reliability in the assessment of physiotherapy students. https://app.cochat.ai/papers/10.1186/s12909-022-03231-y
The paper
Gittinger, F. P., Lemos, M., Neumann, J. L., Förster, J., Dohmen, D., Berke, B., Olmeo, A., Lucas, G., & Jonas, S. M. (2022). Interrater reliability in the assessment of physiotherapy students. BMC Medical Education. https://doi.org/10.1186/s12909-022-03231-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.