The DOI matches its Crossref record; 29 of the 31 references it cites were found in a registry or an index.
COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study
Open accessPublished version of a preprint
Abstract
Background: Scores on an outcome measurement instrument depend on the type and settings of the instrument used, how instructions are given to patients, how professionals administer and score the instrument, etc. The impact of all these sources of variation on scores can be assessed in studies on reliability and measurement error, if properly designed and analyzed. The aim of this study was to develop standards to assess the quality of studies on reliability and measurement error of clinician-reported outcome measurement instruments, performance-based outcome measurement instrument, and laboratory values. Methods: We conducted a 3-round Delphi study involving 52 panelists. Results: Consensus was reached on how a comprehensive research question can be deduced from the design of a reliability study to determine how the results of a study inform us about the quality of the outcome measurement instrument at issue. Consensus was reached on components of outcome measurement instruments, i.e. the potential sources of variation. Next, we reached consensus on standards on design requirements ( n = 5), standards on preferred statistical methods for reliability ( n = 3) and measurement error ( n = 2), and their ratings on a four-point scale. There was one term for a component and one rating of one standard on which no consensus was reached, and therefore required a decision by the steering committee. Conclusion: We developed a tool that enables researchers with and without thorough knowledge on measurement properties to assess the quality of a study on reliability and measurement error of outcome measurement instruments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
Confirmatory factor analysis for applied research
Brown, T. A. (2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,621
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465
Self-citationMokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9606-8 · Cited by 4,260
Self-citationCOSMIN guideline for systematic reviews of patient-reported outcome measures
Prinsen, C. A. C., Mokkink, L. B., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1798-3 · Cited by 3,345
Self-citationThe Delphi technique: myths and realities
Powell, C. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02537.x · Cited by 2,611
Research works printed but not linked: 2
- Shavelson RJ, Webb NM. Generalizability Theory. A primer. Measurement Methods for the Social Science, vol. 1. Newbury Park: Sage Publishing; 1991.
- CANCHILD Centre for disability research, I.o.a.h.s., McMaster University. Outcome measures rating form guideline. 2004[31/8/2017]; Available from: https://www.canchild.ca/system/tenon/assets/attachments/000/000/372/original/measrate.pdf?license=yes.
Cited by
Sorted by citations
COSMIN guideline for systematic reviews of patient-reported outcome measures version 2.0
Mokkink, L. B., Elsman, E. B. M., & Terwee, C. B. (2024) · Quality of Life Research · doi:10.1007/s11136-024-03761-6 · Cited by 562
“…other groups in conducting their reviews [e.g. 18–20], as well as discussions in two Delphi studies in which we developed the COSMIN Risk of Bias tool for assessing reliability and measurement error [21], and the PRISMA-COSMIN for Outcome Measurement Instruments (OMIs) reporting guideline [22].”
InfluentialSelf-citationA systematic review of AI literacy scales
Lintner, T. (2024) · npj Science of Learning · doi:10.1038/s41539-024-00264-4 · Cited by 292
“…based Standards for the selection of health Measurement INstruments (COSMIN) 52 – 54 for the self-report scales, and additionally the COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments 55 for the performance-based scales.”
Scataglini, S., Abts, E., Bocxlaer, C. V., den Bussche, M. V., Meletani, S., & Truijen, S. (2024) · Sensors · doi:10.3390/s24113686 · Cited by 207
“Within the included studies, nineteen [31–36, 38”
MethodWatt, J. A., Veroniki, A. A., Tricco, A. C., & Straus, S. E. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01228-7 · Cited by 163
Casey, P., Alasmar, M., McLaughlin, J., Ang, Y., McPhee, J., Heire, P., & Sultan, J. (2022) · Journal of Cachexia, Sarcopenia and Muscle · doi:10.1002/jcsm.13041 · Cited by 164
“A score was determined by taking the lowest rating of each criterion and defined to be poor, fair, good, or excellent.(21) Methodological quality was scored independently between reviewers, and agreement assessed using Cohen’s kappa coefficient where a score of >0.”
MethodDelgado, A. H. S., Sauro, S., Lima, A. F., Loguércio, A. D., Bona, Á. D., Mazzoni, A., Collares, F. M., Staxrud, F., Ferracane, J. L., Tsoi, J. K. H., et al. (2022) · Journal of Dentistry · doi:10.1016/j.jdent.2022.104350 · Cited by 155
Stefana, A., Damiani, S., Granziol, U., Provenzani, U., Solmi, M., Youngstrom, E. A., & Fusar‐Poli, P. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1494261 · Cited by 220
Whittaker, J. L., Culvenor, A. G., Juhl, C. B., Berg, B., Bricca, A., Filbay, S. R., Holm, P. M., Macri, E. M., Urhausen, A. P., Ardern, C. L., et al. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2022-106299 · Cited by 139
Swan, K., Speyer, R., Scharitzer, M., Farneti, D., Brown, T., Woisard, V., & Cordier, R. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1225850 · Cited by 157
“Measurement (SEM), Smallest Detectable Change (SDC), Limits of Agreement (LoA) or Coefficient of Variation (CV) (Mokkink et al., 2020) Dichotomous/nominal/ordinal scores: the percentage”
BackgroundMethodInfluentialElsman, E. B. M., Mokkink, L. B., Terwee, C. B., Beaton, D., Gagnier, J. J., Tricco, A. C., Baba, A., Butcher, N. J., Smith, M., Hofstetter, C., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111422 · Cited by 126
Self-citationElsman, E. B. M., Mokkink, L. B., Terwee, C. B., Beaton, D., Gagnier, J. J., Tricco, A. C., Baba, A., Butcher, N. J., Smith, M., Hofstetter, C., et al. (2024) · Quality of Life Research · doi:10.1007/s11136-024-03634-y · Cited by 103
“An OMI consists of a set of components and phases, i.e., ‘equipment’, ‘preparatory actions’, ‘collection of raw data’, ‘data processing and storage’, and ‘assignment of the score’ [57].”
InfluentialSelf-citationMokkink, L. B., Eekhout, I. E., Boers, M., van der Vleuten, C. P. M., & de Vet, H. C. W. (2023) · Patient Related Outcome Measures · doi:10.2147/prom.s398886 · Cited by 90
“Often in the clinical field, the object of measurement represents patients, and we use patients as the population of interest in this paper, but it can also be proxies, healthcare professionals, or body structures (eg, joints or lesions).(2) Figure 4 Nested inter-rater reliability design where the raters are nested in…”
InfluentialSelf-citationFarhan, M., Wang, J. Z., Bray, P., Burns, J., & Cheng, T. L. (2021) · Journal of Foot and Ankle Research · doi:10.1186/s13047-020-00442-8 · Cited by 100
“COSMIN is the gold standard for patient reported outcome measurement evaluation, however more recently it has been applied to clinician-rated assessments [17, 20, 21].”
MethodEvangelista, M. C., Monteiro, B. P., & Steagall, P. V. M. (2021) · Pain · doi:10.1097/j.pain.0000000000002474 · Cited by 80
Development and validation of combined symptom‐medication scores for allergic rhinitis*
Sousa‐Pinto, B., Azevedo, L. F., Jutel, M., Agache, I. O., Canonica, G. W., Czarlewski, W., Papadopoulos, N. G., Bergmann, K., Devillier, P., Laune, D., et al. (2021) · Allergy · doi:10.1111/all.15199 · Cited by 77
Tribology properties of additively manufactured Ti6Al4V alloy after heat treatment
Huang, J., Zhang, W., Fang, W., & Yi, Y. (2023) · Tribology International · doi:10.1016/j.triboint.2023.108485 · Cited by 74
Interobserver variability studies in diagnostic imaging: a methodological systematic review
Quinn, L., Tryposkiadis, K., Deeks, J. J., de Vet, H. C. W., Mallett, S., Mokkink, L. B., Takwoingi, Y., Taylor‐Phillips, S., & Sitch, A. J. (2023) · British Journal of Radiology · doi:10.1259/bjr.20220972 · Cited by 71
Self-citationElsman, E. B. M., Butcher, N. J., Mokkink, L. B., Terwee, C. B., Tricco, A. C., Gagnier, J. J., Aiyegbusi, O. L., Barnett, C., Smith, M., Moher, D., et al. (2022) · Systematic Reviews · doi:10.1186/s13643-022-01994-5 · Cited by 75
“Based on previous experiences [45, 11], it is anticipated that at the most 50% of the invited persons will complete at least one round.”
BackgroundSelf-citationNuwailati, R., Bobos, P., Drangsholt, M. T., & Curatolo, M. (2022) · Scandinavian Journal of Pain · doi:10.1515/sjpain-2021-0149 · Cited by 66
Prill, R., Walter, M., Królikowska, A., & Becker, R. (2021) · Sensors · doi:10.3390/s21248221 · Cited by 65
“A COSMIN Risk of Bias tool was used to examine the quality in a systematic and transparent manner [28].”
Prill, R., Królikowska, A., de Girolamo, L., Becker, R., & Karlsson, J. (2023) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-023-07442-8 · Cited by 68
Menahemi-Falkov, M., Breitenstein, C., Pierce, J. E., Hill, A. J., O’Halloran, R., & Rose, M. L. (2021) · Disability and Rehabilitation · doi:10.1080/09638288.2021.1955303 · Cited by 66
“…the variability between two assessments into account when the SEm is used for calculating critical change scores such as the MDC 90 cut-off value [42], preferably the “absolute agreement” “single measure” variant of the intra-class correlation coefficient/ICC, the generalisability coefficient or…”
Molina‐García, P., Notbohm, H. L., Schumann, M., Argent, R., Hetherington‐Rauth, M., Stang, J. S., Bloch, W., Cheng, S., Ekelund, U., Sardinha, L. B., et al. (2022) · Sports Medicine · doi:10.1007/s40279-021-01639-y · Cited by 54
Urhausen, A. P., Berg, B., Øiestad, B. E., Whittaker, J. L., Culvenor, A. G., Crossley, K. M., Juhl, C. B., & Risberg, M. A. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2022-105498 · Cited by 48
Beshara, P., Anderson, D., Pelletier, M. H., & Walsh, W. R. (2021) · Sensors · doi:10.3390/s21248186 · Cited by 40
“The authors acknowledge an appropriate time interval depends on the stability of the construct (COSMIN item one), and the target population [88].”
BackgroundMethodDevices measuring transepidermal water loss: A systematic review of measurement properties
Klotz, T., Ibrahim, A., Maddern, G. J., Caplash, Y., & Wagstaff, M. J. D. (2022) · Skin Research and Technology · doi:10.1111/srt.13159 · Cited by 43
Eiger, B., Errebo, M., Straszek, C. L., & Vægter, H. B. (2022) · Scandinavian Journal of Pain · doi:10.1515/sjpain-2021-0200 · Cited by 41
Scialpi, A., Mignolli, E., Vito, C. D., Berardi, A., Tofani, M., Valente, D., & Galeoto, G. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159132 · Cited by 42
Operationalising treatment success in aphasia rehabilitation
Breitenstein, C., Hilari, K., Menahemi-Falkov, M., Rose, M. L., Wallace, S. J., Brady, M., Hillis, A. E., Kiran, S., Szaflarski, J. P., Tippett, D. C., et al. (2022) · Aphasiology · doi:10.1080/02687038.2021.2016594 · Cited by 37
“An alternate option is that parallel versions of a test may be administered for repeated assessments to disentangle changes in 'true' scores from measurement error (e.g., using bi-factor models; Mokkink, Boers, van der Vleuten, Bouter, et al., 2020, p. 31).”
BackgroundMethodBerg, B., Urhausen, A. P., Øiestad, B. E., Whittaker, J. L., Culvenor, A. G., Roos, E. M., Crossley, K. M., Juhl, C. B., & Risberg, M. A. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2022-105510 · Cited by 43
Stallwood, E., Monsour, A., Rodrigues, C., Monga, S., Terwee, C. B., Offringa, M., & Butcher, N. J. (2020) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2020.10.009 · Cited by 39
Self-citationLabib, K., Roje, R., Bouter, L. M., Widdershoven, G. A. M., Evans, N., Marušić, A., Mokkink, L. B., & Tijdink, J. K. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00322-9 · Cited by 38
“This explanation is supported by the fact that two recent Delphi studies using the same means of reporting their response rates had similar figures to ours (Haven et al., 2020; Mokkink et al., 2020).”
ResultSelf-citationGuidelines for the development and validation of patient-reported outcome measures: a scoping review
Jung, A., Challoumas, D., Pagels, L., Armijo-Olivo, S. L., Braun, T., & Luedtke, K. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112681 · Cited by 36
“21 As a result, many of their methods are considered the current best reference standards in instrument development and (evaluation of) quality of measurement properties.”
Khaleva, E., Rattu, A., Brightling, C. E., Bush, A., BOURDIN, A. A., Bossios, A., Chung, K. F., Chaudhuri, R., Coleman, C., Djukanović, R., et al. (2023) · ERJ Open Research · doi:10.1183/23120541.00444-2022 · Cited by 36
de Arruda, G. T., Terwee, C. B., Elsman, E. B. M., Ávila, M. A., Gagnier, J. J., Mokkink, L. B., PROM Reporting Group, Dibai-Filho, A. V., Firth, A. D., Mehdipour, A., et al. (2025) · Quality of Life Research · doi:10.1007/s11136-025-03949-4 · Cited by 33
“In our experience, methods guidelines and reporting guidelines need to be updated regularly, for example, the COSMIN Risk of bias checklist [5, 12, 19, 20] and the COSMIN guideline on conducting systematic reviews [1, 5] have been updated over the years, as have the CONSORT statement [21, 22] and…”
Self-citationTomacheuski, R. M., Monteiro, B. P., Evangelista, M. C., Luna, S. P. L., & Steagall, P. V. M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280830 · Cited by 32
“Adapted from the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) [38, 41, 45, 47].”
MethodInfluentialStarmer, H. M., Cherry, M. G., Patterson, J. M., Young, B., & Fleming, J. C. (2022) · Lymphatic Research and Biology · doi:10.1089/lrb.2021.0100 · Cited by 32
“…for each tool was evaluated by one reviewer (H.S.) using the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) risk of bias checklist for reliability studies 16 and the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) 17 for validation studies.”
Varcin, F., & Boocock, M. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103332 · Cited by 36
Elastography in the assessment of the Achilles tendon: a systematic review of measurement properties
Mifsud, T., Gatt, A., Micallef‐Stafrace, K., Chockalingam, N., & Padhiar, N. (2023) · Journal of Foot and Ankle Research · doi:10.1186/s13047-023-00623-1 · Cited by 31
van Exter, S. H., Koenders, N., van der Wees, P. J., & van den Berg, M. G. A. (2024) · Age and Ageing · doi:10.1093/ageing/afae113 · Cited by 35
“The COSMIN Risk of Bias tool [17] was used to critically appraise studies reporting on reliability and measurement error.”
MethodD’Agostino, M. A., Beaton, D., Maxwell, L. J., Cembalo, S. M., Hoens, A. M., Hofstetter, C., Zăbălan, C., Bird, P., Christensen, R., de Wit, M. P. T., et al. (2021) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2021.08.004 · Cited by 29
Self-citationConnor-Davidson Resilience Scale: a systematic review psychometrics properties using the COSMIN
Nia, H. S., Sánchez‐Teruel, D., Froelicher, E. S., Hejazi, S. S., Hosseini, L., Fomani, F. K., Moshtagh, M., Mollaei, F., Goudarzian, A. H., & Babaei, A. (2024) · Annals of Medicine and Surgery · doi:10.1097/ms9.0000000000001968 · Cited by 37
“In other words, assessing the measurement error in each psychometric study can guide the researchers to decide about the level of trust they can place in the study fi ndings and whether the parameters that where estimated are systematically not over-or underestimated [101] .”
Cancino, E. E., Rice, K., Abichandani, D., & Falla, D. (2022) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-022-05066-6 · Cited by 31
“The Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) initiative established a taxonomy of measurement properties which covers the following domains: reliability, validity, and responsiveness [16].”
BackgroundMethodInfluentialSchiltz, H. K., Williams, Z. J., Zheng, S., Kaplan‐Kahn, E. A., Morton, H. E., Rosenau, K. A., Nicolaidis, C., Sturm, A., Lord, C., & PROMnet, A. (2024) · Autism Research · doi:10.1002/aur.3114 · Cited by 30
“Notably, a number of tools such as the COSMIN extensions (Gagnier et al., 2021; Mokkink et al., 2016, 2018, 2020; Prinsen et al., 2016, 2018; Terwee et al., 2018) and the ConPsy checklist (Vitoratou et al., 2024) are also available to guide the development and evaluation of psychometric instruments…”
MethodUljarević, M., Spackman, E. K., Whitehouse, A., Frazier, T. W., Billingham, W., Condron, P., Hardan, A. Y., & Leekam, S. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2023.102286 · Cited by 27
Waiteman, M. C., Garcia, M. C., Briani, R. V., Norte, G. E., Glaviano, N. R., de Azevedo, F. M., & Bazett-Jones, D. M. (2023) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2023.12045 · Cited by 29
MacRae, J. M., Harasemiw, O., Lightfoot, C. J., Thompson, S., Wytsma-Fisher, K., Koufaki, P., Bohm, C. J., & Wilkinson, T. J. (2023) · Clinical Kidney Journal · doi:10.1093/ckj/sfad170 · Cited by 28
“…measurement property ( very good/adequate/doubtful/inadequate ) and reached consensus on each item in the checklist, as it applied to the individual study; the rankings derived from this then informed the overall RoB score using the COSMIN RoB tool [ 15 , 17 ] ( Supplementary Material 3 and 4 ) .”
BackgroundMethodInfluentialElsman, E. B. M., Mokkink, L. B., Terwee, C. B., Beaton, D., Gagnier, J. J., Tricco, A. C., Baba, A., Butcher, N. J., Smith, M., Hofstetter, C., et al. (2024) · Journal of Patient-Reported Outcomes · doi:10.1186/s41687-024-00727-7 · Cited by 27
“An OMI consists of a set of components and phases, i.e., ‘equipment’, ‘preparatory actions’, ‘collection of raw data’, ‘data processing and storage’, and ‘assignment of the score’ [57].”
BackgroundMethodInfluentialSelf-citationScataglini, S., Fontinovo, E., Khafaga, N., Khan, M. U., Khan, M. F., & Truijen, S. (2025) · Sensors · doi:10.3390/s25175513 · Cited by 25
“The Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) [26,27] was used for methodological quality appraisal.”
InfluentialBarbosa, G. M., Calixtre, L. B., Fialho, H. R. F., Locks, F., & Kamonseki, D. H. (2023) · Brazilian Journal of Physical Therapy · doi:10.1016/j.bjpt.2023.100575 · Cited by 32
“The methodological quality of inter-session SEM and the MDC 95 was very good, 9,10 and no evidence was established for measurement error because the MIC have not been de fi ned.”
BackgroundMethodInfluentialA practical guide for selecting continuous monitoring wearable devices for community-dwelling adults
Lu, J. K., Wang, W., Goh, J., & Maier, A. B. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e33488 · Cited by 25
“The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) Risk of Bias tool was integrated to ensure a procedure for evaluating technical performance and validation was included [13].”
BackgroundMethodInterpreting Change in Disorders of Consciousness Using the Coma Recovery Scale-Revised
Weaver, J., Cogan, A. M., Kozlowski, A. J., Grady-Dominguez, P., O’Brien, K. A., Bodien, Y. G., Graham, J. E., Aichele, S., Ford, P., Kot, T., et al. (2024) · Journal of Neurotrauma · doi:10.1089/neu.2023.0567 · Cited by 24
Within-subject variation of HbA1c: A systematic review and meta-analysis
Gough, A., Sitch, A. J., Ferris, E., & Marshall, T. P. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289085 · Cited by 22
Atigossou, O. L. G., Honado, A. S., Routhier, F., & Flamand, V. H. (2021) · Assistive Technology · doi:10.1080/10400435.2021.2010149 · Cited by 22
“The COnsensus-based Standards for the Selection of Health Measurement INstruments Risk of Bias checklist (COSMINRoB) was used to critically appraise the risk of bias of each psychometric data reported in each of the included studies (Mokkink et al., 2020, 2018).”
MethodThron, M., Düking, P., Ruf, L., Härtel, S., Wöll, A., & Altmann, S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0296866 · Cited by 22
Oliveira, H. C., Neto, D. H., Carvalho, S. D. L., de Barros, R. D. C. L., Neves, M. L. D. S., Andrechuk, C. R. S., Alexandre, N. M. C., Ribeiro, P. A. B., & Rodrigues, R. C. M. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02340-z · Cited by 27
Yu, H., Sun, C., Sun, B., Chen, X., & Tan, Z. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182412912 · Cited by 21
“For analyses where I2 was less than 50%, a fixed-effects model was used; if I2 was more than 50%, a random-effects model was employed [53].”
InfluentialAutomatic market research of mobile health apps for the self‐management of allergic rhinitis
Antó, A., Sousa‐Pinto, B., Czarlewski, W., Pfaar, O., Bosnic-Anticevich, S. Z., Klimek, L., Matricardi, P. M., Tripodi, S., Fonseca, J., Antó, J. M., et al. (2022) · Clinical & Experimental Allergy · doi:10.1111/cea.14135 · Cited by 21
The empty pelvis syndrome: a core data set from the PelvEx collaborative
Collaborative, P., Mirnezami, A. H., Drami, I., Glyn, T., Sutton, P. A., Tiernan, J. P., Behrenbruch, C., Guerra, G., Waters, P. S., Woodward, N., et al. (2024) · British journal of surgery · doi:10.1093/bjs/znae042 · Cited by 22
“Two studies were identified that calibrated the size of the empty pelvic cavity following PE to measure the size of appropriate breast implants for pelvic filling and reduction of dead space; however, these were not feasible and were inadequate using the COSMIN risk of bias tool 13,45,89”
MethodElsman, E. B. M., Mokkink, L. B., Terwee, C. B., Beaton, D., Gagnier, J. J., Tricco, A. C., Baba, A., Butcher, N. J., Smith, M., Hofstetter, C., et al. (2024) · Health and Quality of Life Outcomes · doi:10.1186/s12955-024-02256-9 · Cited by 34
“The COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of OMIs can be used for any type of OMI [57]”
InfluentialSelf-citationPettit, M. H., Kanavathy, S., McArthur, N., Weiss, O., & Khanduja, V. (2022) · The Journal of Arthroplasty · doi:10.1016/j.arth.2022.05.029 · Cited by 21
Zhu, Z., Xing, W., Wen, H., Sun, Y., So, W. K. W., Lizarondo, L. M., Peng, J. W., & Hu, Y. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057215 · Cited by 20
Identifying and appraising outcome measures for severe asthma: a systematic review
Rattu, A., Khaleva, E., Brightling, C. E., Dahlén, S., Bossios, A., Fleming, L., Chung, K. F., Melén, E., Djukanović, R., Chaudhuri, R., et al. (2022) · European Respiratory Journal · doi:10.1183/13993003.01231-2022 · Cited by 20
Gough, A., Sitch, A. J., Ferris, E., & Marshall, T. P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0304961 · Cited by 22
Update on the psychometric properties for the Italian version of the Aachen Aphasia Test (IT-AAT)
Luzzatti, C., Bleser, R. D., Scola, I., Frustaci, M., & Willmes, K. (2022) · Aphasiology · doi:10.1080/02687038.2022.2037501 · Cited by 19
Walsh, C. B., Cahalan, R., Hinman, R. S., & Sullivan, K. O. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274349 · Cited by 19
“The methodological quality of each of the included studies was evaluated by two independent authors (CBW & KOS) using the COSMIN risk of bias checklist and scores were determined by consensus [34].”
InfluentialPsychometric properties of functional postural control tests in children: A systematic review
Johnson, C., Hallemans, A., Goetschalckx, M., Meyns, P., Rameckers, E. A. A., Klingels, K., & Verbecque, E. (2023) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2022.101729 · Cited by 23
Lunny, C., Veroniki, A. A., Hutton, B., White, I. R., Higgins, J., Wright, J. M., Kim, J. Y., Thirugnanasampanthar, S. S., Siddiqui, S., Watt, J. A., et al. (2022) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-111944 · Cited by 19
A systematic review of validated assessments methods for head and neck lymphedema
Arends, C. R., Lindhout, J. E., van der Molen, L., Wilthagen, E. A., van den Brekel, M. W. M., & Stuiver, M. M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-07841-0 · Cited by 19
“For studies on clinician-rated instruments, we used the COSMIN Risk of Bias tool for assessing the quality of studies on reliability and measurement error of outcome measurement instruments [31].”
MethodAssessing parental emotion regulation in the context of parenting: A systematic review
Zhang, X., Beatty, A., Abela, K. R., Melo, M. F., Kenny, M., Atkinson, L. R., & González, A. (2023) · Developmental Review · doi:10.1016/j.dr.2023.101092 · Cited by 27
AlDahas, A., Heneghan, N. R., Althobaiti, S., Deane, J. A., Rushton, A., & Falla, D. (2024) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-023-07111-4 · Cited by 25
“…reliability and measurement error of outcome measurement instruments as well as the COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs) [13, 14] and is reported in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) [15].”
MethodInfluentialAtaman, R., Alhasani, R., Auneau-Enjalbert, L., Quigley, A., Michael, H. U., & Ahmed, S. (2024) · Journal of Patient-Reported Outcomes · doi:10.1186/s41687-024-00743-7 · Cited by 17
“The overall rating of the methodological quality for a measurement property was based on the worst item rating [16, 24].”
MethodZhou, J., Yao, Q., Han, R., Bock, P. D., Vassard-Yu, G., Hallemans, A., & Laer, L. V. (2025) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2025.03.001 · Cited by 19
Geppert, J., Auguste, P., Asgharzadeh, A., Ghiasvand, H., Patel, M., Brown, A., Jayakody, S., Helm, E., Todkill, D., Madan, J., et al. (2025) · Health Technology Assessment · doi:10.3310/jytw8921 · Cited by 22
Ataman, R., Thomas, A., Roberge‐Dao, J., McKerral, M., Auger, C., Wittich, W., Talla, P. K., Boychuck, Z., & Ahmed, S. (2023) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2022.12.196 · Cited by 19
Measures of financial toxicity in cancer survivors: a systematic review
Thomy, L. B., Crichton, M., Jones, L. V., Yates, P. M., Hart, N. H., Collins, L. G., & Chan, R. J. (2024) · Supportive Care in Cancer · doi:10.1007/s00520-024-08601-4 · Cited by 22
“COSMIN criteria requires evaluation of relevance, comprehensive-ness, and comprehensibility in the assessment of content validity [15]].”
BackgroundInfluentialClinically relevant estimation of minimal number of trials for the uncontrolled manifold analysis
Piscitelli, D., Buttram, A., Abernathy, K., Canelón, J., Knighten, D., & Solnik, S. (2024) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2024.112195 · Cited by 17
Williams, L. J., Loetscher, T., Hillier, S., Hreha, K. P., Jones, J., Bowen, A., & Kernot, J. (2024) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2024.2346212 · Cited by 16
“This study is the first to report on the quality of the psychometric evidence for spatial neglect tools, as standards for evaluating the evidence such as COSMIN (Mokkink et al., 2020) have only recently been developed.”
InfluentialMárquez, M. A., Speroni, A., Galeoto, G., Ruotolo, I., Sellitto, G., Tofani, M., Gonzalez-Bernal, J. J., & Berardi, A. (2022) · Spinal Cord Series and Cases · doi:10.1038/s41394-022-00492-z · Cited by 16
Zapata-Figueroa, V., & Ortiz‐Corredor, F. (2022) · Occupational Therapy International · doi:10.1155/2022/9980523 · Cited by 16
“From the perspective of the COSMIN standards for assessing the quality of studies on reliability and measurement error, the BBT is an example of a performance-based outcome measurement instrument (PerFOMs) in which a professional instructs a patient to comply with a specific manual task [30].”
MethodFearn, N., Llanos, C., Dylke, E. S., Stuart, K. E., & Kilbreath, S. L. (2022) · Journal of Cancer Survivorship · doi:10.1007/s11764-022-01278-w · Cited by 17
Lu, J. K., Sijm, M., Janssens, G. E., Goh, J., & Maier, A. B. (2023) · GeroScience · doi:10.1007/s11357-023-00815-4 · Cited by 18
“The user manual of the COSMIN Risk of Bias tool was used to guide the reporting of validation outcomes [12].”
MethodHonvo, G., Sabico, S., Veronese, N., Bruyère, O., Rizzoli, R., Thiyagarajan, J. A., Mikton, C., Diaz, T., Cooper, C. C., & Reginster, J. (2023) · Age and Ageing · doi:10.1093/ageing/afad139 · Cited by 21
Development, refinement, and validation of an equine musculoskeletal pain scale
Auer, U., Kelemen, Z., Vogl, C., von Ritgen, S., Haddad, R., Borda, L. T., Gabmaier, C., Breteler, J., & Jenner, F. (2024) · Frontiers in Pain Research · doi:10.3389/fpain.2023.1292299 · Cited by 16
The Long COVID Symptoms and Severity Score: Development, Validation, and Application
Ye, G., Zhu, Y., Bao, W., Zhou, H., Lai, J., Zhang, Y., Xie, J., Ma, Q., Luo, Z., Ma, S., et al. (2024) · Value in Health · doi:10.1016/j.jval.2024.04.009 · Cited by 16
Reproducibility of Microwave Breast Imaging: Analysis of Regular Scans of a Group of Volunteers
Wang, Z., Mojabi, P., Price, S., Lasemiimeni, Z., Besler, B. C., Bourqui, J., Deutscher, D., Besler, B. A., Shen, H., Docktor, B., et al. (2024) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2024.3512572 · Cited by 16
“The COSMIN Risk of Bias Tool was developed to assess the quality of studies addressing reliability and measurement variability [22], combining elements for an effective research question with 8-9 standards to assess the quality of a study.”
González-Seguel, F., Pinto-Concha, J. J., Ríos-Castro, F., Silva-Gutiérrez, A., Camus-Molina, A., Mayer, K. P., & Parry, S. M. (2021) · Archives of Rehabilitation Research and Clinical Translation · doi:10.1016/j.arrct.2021.100142 · Cited by 16
Egerton, T., Lawford, B. J., Campbell, P. K., Plinsinga, M. L., Spiers, L., Mackenzie, D., Graham, B., Mills, K. A. G., Eyles, J. P., Knox, G., et al. (2021) · JMIR Formative Research · doi:10.2196/32627 · Cited by 16
Reply to the concerns raised by McKenna and Heaney about COSMIN
Mokkink, L. B., Terwee, C. B., de Vet, H. C. W., Bouter, L. M., Alonso, J., & Patrick, D. L. (2021) · Journal of Medical Economics · doi:10.1080/13696998.2021.1948231 · Cited by 18
Self-citationThe accuracy of Apple Watch measurements: a living systematic review and meta-analysis
Lambe, R., Baldwin, M., O’Grady, B., Schumann, M., Caulfield, B., & Doherty, C. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02238-1 · Cited by 17
Aledda, S., Galeoto, G., Fabbrini, G., Lucibello, L., Tofani, M., Conte, A., & Berardi, A. (2023) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2022.2161648 · Cited by 15
“The review followed the 27-item Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist [19] and COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodology for systematic reviews of patient-reported outcome measures (PROMs) [10].”
MethodInfluentialZhou, E. F. M., Fu, S. N., Huang, C., Huang, X. P., & Wong, A. Y. L. (2023) · BMC Oral Health · doi:10.1186/s12903-023-03558-y · Cited by 19
“The methodological quality of the included studies was rated as “very good, adequate, doubtful, or inadequate” using the “worst-score counts” principle [31, 32].”
BackgroundMethodBudgett, J., Sommerlad, A., Kupeli, N., Zabihi, S., Olsen, A., & Cooper, C. (2023) · Dementia · doi:10.1177/14713012231222309 · Cited by 16
“In this review, we follow the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) guidelines (Mokkink et al., 2020), a comprehensive checklist designed to assess the psychometric properties and methodological quality of outcome measures.”
MethodInfluentialZhou, E. F. M., Wong, A. Y. L., Zheng, Y., Lam, K. H. S., & Fu, S. N. (2024) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2023.12.021 · Cited by 15
Trul-Kreuze, I. A., Akkerman, M., Kieboom, E. A. M., Nieuwenhuis, M. K., Houdijk, H., & Bongers, B. C. (2024) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2024.02.717 · Cited by 15
Smith, M. G., Farrar, L. C., Gibson, R. J., Russo, R. N., & Harvey, A. R. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15535 · Cited by 15
Kwok, E. Y. L., Rosenbaum, P., Thomas‐Stonell, N., & Cunningham, B. J. (2021) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.12603 · Cited by 17
Skibsted, C. V., Jensen, B. T., Juul, T., & Kristensen, H. Ø. (2022) · Colorectal Disease · doi:10.1111/codi.16202 · Cited by 14
“To determine the rating, the corresponding COSMIN Risk of Bias box was completed, with the lowest rating in the box deciding the overall rating in concordance with the user manual [30].”
MethodCleveland, M. R., Baute, R., Clindaniel, C., Hertz, L., Pond, R., & Centers, G. I. (2023) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003182 · Cited by 8
“Study procedures were designed following the Consensus-based Standards for the Selection of Health Measurement Instruments risk of bias tool (11) and reported following the Guidelines for Reporting Reliability and Agreement Studies (12).”
MethodTest-retest reliability of joint position sense in the elbow among healthy adults
Rider, J. V., & Valdes, K. A. (2023) · Journal of Hand Therapy · doi:10.1016/j.jht.2023.08.015 · Cited by 15
Metadata
- Authors
- Mokkink, L. B., Boers, M., van der Vleuten, C. P. M., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B.
- Affiliations
- Amsterdam Public Health (NL)
- Universitat Pompeu Fabra (ES)
- University of Washington (US)
- Corresponding author
- Mokkink, L. B.
- Venue
- BMC Medical Research Methodology
- Published
- 2020-12
- Volume
- 20
- Issue
- 1
- Pages
- 293-293
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC7712525
- OpenAlex
- W3107903990
- Cited by
- 588
- References
- 31
- Citations by year
- 2026: 106 · 2025: 121 · 2024: 149 · 2023: 101 · 2022: 72 · 2021: 37 · 2020: 2
- FWCI
- 32.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- VU Research Portal (publishedVersion, other-oa)
- Pure Amsterdam UMC (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (submittedVersion)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (91617098)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- COSMIN, risk of bias, reliability statistics, measurement error, outcome measurement instruments, clinician-reported outcomes, performance-based outcome measures, laboratory parameters, study quality assessment, Delphi consensus, measurement properties
- MeSH terms
- Delphi Technique, Humans, Reproducibility of Results, Bias, Consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
