The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist
Open accessAbstract
BACKGROUND: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). METHODS: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. RESULTS: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions. CONCLUSIONS: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Statistical Methods for Rates and Proportions.
Everitt, B. S., & Fleiss, J. L. (1981) · Biometrics · doi:10.2307/2530193 · Cited by 8,990
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465
Self-citationMokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9606-8 · Cited by 4,260
Self-citationMokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., Bouter, L. M., & de Vet, H. C. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-22 · Cited by 1,834
Self-citationToutenburg, H. (1974) · Biometrische Zeitschrift · doi:10.1002/bimj.19740160814 · Cited by 1,538
Kappa coefficients in medical research
Chmura Kraemer, H., Periyakoil, V. S., & Noda, A. (2002) · Statistics in Medicine · doi:10.1002/sim.1180 · Cited by 527
A One-Way Components of Variance Model for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529465 · Cited by 478
Development of EMPRO: A Tool for the Standardized Assessment of Patient-Reported Outcome Measures
Valderas, J. M., Ferrer, M., Mendívil, J., Garin, O., Rajmil, L., Herdman, M., & Alonso, J. (2008) · Value in Health · doi:10.1111/j.1524-4733.2007.00309.x · Cited by 228
Self-citationThe dependence of Cohen's kappa on the prevalence does not matter
Vach, W. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.021 · Cited by 182
Cited by
Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., Bouter, L. M., & de Vet, H. C. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-22 · Cited by 1,834
Self-citationEpstein, J., Santo, R. M., & Guillemin, F. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.021 · Cited by 990
Mokkink, L. B., Prinsen, C. A. C., Bouter, L. M., Vet, H. C. W. D., & Terwee, C. B. (2016) · Brazilian Journal of Physical Therapy · doi:10.1590/bjpt-rbf.2014.0143 · Cited by 895
Self-citationReeve, B. B., Wyrwich, K. W., Wu, A. W., Velikova, G., Terwee, C. B., Snyder, C. F., Schwartz, C., Revicki, D. A., Moinpour, C. M., McLeod, L. D., et al. (2013) · Quality of Life Research · doi:10.1007/s11136-012-0344-y · Cited by 845
Self-citationMoore, J. L., Potter, K., Blankshain, K., Kaplan, S. L., O'Dwyer, L. C., & Sullivan, J. E. (2018) · Journal of Neurologic Physical Therapy · doi:10.1097/npt.0000000000000229 · Cited by 363
Metadata
- Authors
- Mokkink, L. B., Terwee, C. B., Gibbons, E., Stratford, P. W., Alonso, J., Patrick, D. L., Knol, D. L., Bouter, L. M., & de Vet, H. C.
- Venue
- BMC Medical Research Methodology
- Published
- 2010-09-22
- Volume
- 10
- Issue
- 1
- Pages
- 82-82
- ISSN
- 1471-2288 (ISSN-L)
- OpenAlex
- W2036060563
- Cited by
- 285
- References
- 13
- FWCI
- 11.91
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist. https://app.cochat.ai/papers/10.1186/1471-2288-10-82
- The paper
- Mokkink, L. B., Terwee, C. B., Gibbons, E., Stratford, P. W., Alonso, J., Patrick, D. L., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2010). Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-10-82
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.