The DOI matches its Crossref record; 29 of the 31 references it cites were found in a registry or an index.
COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study
Open accessPublished version of a preprint
Abstract
Background: Scores on an outcome measurement instrument depend on the type and settings of the instrument used, how instructions are given to patients, how professionals administer and score the instrument, etc. The impact of all these sources of variation on scores can be assessed in studies on reliability and measurement error, if properly designed and analyzed. The aim of this study was to develop standards to assess the quality of studies on reliability and measurement error of clinician-reported outcome measurement instruments, performance-based outcome measurement instrument, and laboratory values. Methods: We conducted a 3-round Delphi study involving 52 panelists. Results: Consensus was reached on how a comprehensive research question can be deduced from the design of a reliability study to determine how the results of a study inform us about the quality of the outcome measurement instrument at issue. Consensus was reached on components of outcome measurement instruments, i.e. the potential sources of variation. Next, we reached consensus on standards on design requirements ( n = 5), standards on preferred statistical methods for reliability ( n = 3) and measurement error ( n = 2), and their ratings on a four-point scale. There was one term for a component and one rating of one standard on which no consensus was reached, and therefore required a decision by the steering committee. Conclusion: We developed a tool that enables researchers with and without thorough knowledge on measurement properties to assess the quality of a study on reliability and measurement error of outcome measurement instruments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
Confirmatory factor analysis for applied research
Brown, T. A. (2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,621
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465
Self-citationMokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9606-8 · Cited by 4,260
Self-citationCOSMIN guideline for systematic reviews of patient-reported outcome measures
Prinsen, C. A. C., Mokkink, L. B., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1798-3 · Cited by 3,345
Self-citationThe Delphi technique: myths and realities
Powell, C. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02537.x · Cited by 2,611
COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures
Mokkink, L. B., de Vet, H. C. W., Prinsen, C. A. C., Patrick, D. L., Alonso, J., Bouter, L. M., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-017-1765-4 · Cited by 2,479
Self-citationTerwee, C. B., Prinsen, C. A. C., Chiarotto, A., Westerman, M. J., Patrick, D. L., Alonso, J., Bouter, L. M., de Vet, H. C. W., & Mokkink, L. B. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1829-0 · Cited by 2,348
Self-citationMeasurement in Medicine: A Practical Guide
de Vet, H. C. W., Terwee, C. B., Mokkink, L. B., & Knol, D. L. (2011) · doi:10.1017/cbo9780511996214 · Cited by 2,321
Self-citationGuidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A. P., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M. M., & Streiner, D. L. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.002 · Cited by 2,314
A methodological framework for assessing health indices
Kirshner, B., & Guyatt, G. (1985) · Journal of Chronic Diseases · doi:10.1016/0021-9681(85)90005-0 · Cited by 1,538
(2011) · Amsterdam UMC (VU Amsterdam) - Institutional Repository · Cited by 1,209
Self-citationBrennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed
Kottner, J., Audigé, L., Brorson, S., Donner, A. P., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M. M., & Streiner, D. L. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.01.016 · Cited by 1,032
The development of a quality appraisal tool for studies of diagnostic reliability (QAREL)
Lucas, N. P., Macaskill, P., Irwig, L., & Bogduk, N. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.10.002 · Cited by 376
Walton, M. K., Powers, J. H., Hobart, J. C., Patrick, D. L., Marquis, P., Vamvakas, S., Isaac, M., Molsen, E., Cano, S., & Burke, L. B. (2015) · Value in Health · doi:10.1016/j.jval.2015.08.006 · Cited by 310
Self-citationGoodkin, D. E., Hertsgaard, D., & Seminary, J. (1988) · PubMed · Cited by 263
Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68
Blöch, R., & Norman, G. R. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.703791 · Cited by 248
Qin, S., Nelson, L. M., McLeod, L., Eremenco, S. L., & Coons, S. J. (2018) · Quality of Life Research · doi:10.1007/s11136-018-2076-0 · Cited by 241
Powers, J. H., Patrick, D. L., Walton, M. K., Marquis, P., Cano, S., Hobart, J., Isaac, M., Vamvakas, S., Slagle, A., Molsen, E., et al. (2017) · Value in Health · doi:10.1016/j.jval.2016.11.005 · Cited by 191
Self-citationAudigé, L., Bhandari, M., & Kellam, J. F. (2004) · Acta Orthopaedica Scandinavica · doi:10.1080/00016470412331294445 · Cited by 186
D’Agostino, M. A., Aegerter, P., Jousse‐Joulin, S., Chary-Valckenaere, I., Lecoq, B., Gaudin, P., Brault, I., Schmitz, J., Dehaut, F.-X., Parc, J. M. L., et al. (2008) · Arthritis Care & Research · doi:10.1002/art.24369 · Cited by 98
Development, intra- and inter-rater reliability of the New Zealand Secretion Scale (NZSS)
Miles, A., & Hunting, A. (2018) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2018.1458901 · Cited by 40
The Cleft Aesthetic Rating Scale for 18-Year-Old Unilateral Cleft Lip and Palate Patients
Mulder, F. J., Mosmuller, D. G. M., de Vet, H. C. W., Mouës, C. M., Breugem, C. C., van der Molen, A. B. M., & Griot, J. P. W. D. (2016) · The Cleft Palate-Craniofacial Journal · doi:10.1597/16-123 · Cited by 22
The Impact of Ignoring Measurement Error when Estimating Sample Size for Epidemiologic Studies
Devine, O. J. (2003) · Evaluation & the Health Professions · doi:10.1177/0163278703255232 · Cited by 15
Research works printed but not linked: 2
- Shavelson RJ, Webb NM. Generalizability Theory. A primer. Measurement Methods for the Social Science, vol. 1. Newbury Park: Sage Publishing; 1991.
- CANCHILD Centre for disability research, I.o.a.h.s., McMaster University. Outcome measures rating form guideline. 2004[31/8/2017]; Available from: https://www.canchild.ca/system/tenon/assets/attachments/000/000/372/original/measrate.pdf?license=yes.
Cited by
Sorted by citations
COSMIN guideline for systematic reviews of patient-reported outcome measures version 2.0
Mokkink, L. B., Elsman, E. B. M., & Terwee, C. B. (2024) · Quality of Life Research · doi:10.1007/s11136-024-03761-6 · Cited by 562
“…other groups in conducting their reviews [e.g. 18–20], as well as discussions in two Delphi studies in which we developed the COSMIN Risk of Bias tool for assessing reliability and measurement error [21], and the PRISMA-COSMIN for Outcome Measurement Instruments (OMIs) reporting guideline [22].”
InfluentialSelf-citationA systematic review of AI literacy scales
Lintner, T. (2024) · npj Science of Learning · doi:10.1038/s41539-024-00264-4 · Cited by 292
“…based Standards for the selection of health Measurement INstruments (COSMIN) 52 – 54 for the self-report scales, and additionally the COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments 55 for the performance-based scales.”
Scataglini, S., Abts, E., Bocxlaer, C. V., den Bussche, M. V., Meletani, S., & Truijen, S. (2024) · Sensors · doi:10.3390/s24113686 · Cited by 207
“Within the included studies, nineteen [31–36, 38”
MethodWatt, J. A., Veroniki, A. A., Tricco, A. C., & Straus, S. E. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01228-7 · Cited by 163
Casey, P., Alasmar, M., McLaughlin, J., Ang, Y., McPhee, J., Heire, P., & Sultan, J. (2022) · Journal of Cachexia, Sarcopenia and Muscle · doi:10.1002/jcsm.13041 · Cited by 164
“A score was determined by taking the lowest rating of each criterion and defined to be poor, fair, good, or excellent.(21) Methodological quality was scored independently between reviewers, and agreement assessed using Cohen’s kappa coefficient where a score of >0.”
Method
Metadata
- Authors
- Mokkink, L. B., Boers, M., van der Vleuten, C. P. M., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B.
- Affiliations
- Amsterdam Public Health (NL)
- Universitat Pompeu Fabra (ES)
- University of Washington (US)
- Corresponding author
- Mokkink, L. B.
- Venue
- BMC Medical Research Methodology
- Published
- 2020-12
- Volume
- 20
- Issue
- 1
- Pages
- 293-293
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC7712525
- OpenAlex
- W3107903990
- Cited by
- 588
- References
- 31
- Citations by year
- 2026: 106 · 2025: 121 · 2024: 149 · 2023: 101 · 2022: 72 · 2021: 37 · 2020: 2
- FWCI
- 32.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- VU Research Portal (publishedVersion, other-oa)
- Pure Amsterdam UMC (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (91617098)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- COSMIN, risk of bias, reliability statistics, measurement error, outcome measurement instruments, clinician-reported outcomes, performance-based outcome measures, laboratory parameters, study quality assessment, Delphi consensus, measurement properties
- MeSH terms
- Delphi Technique, Humans, Reproducibility of Results, Bias, Consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
