Evaluation of the methodological quality of systematic reviews of health status measurement instruments
Open accessThe DOI matches its Crossref record; 176 of the 178 references it cites were found in a registry or an index; 3 of the 37 sources its claims cite were checked for support.
Abstract
A systematic review of measurement properties of health-status instruments is a tool for evaluating the quality of instruments. Our aim was to appraise the quality of the review process, to describe how authors assess the methodological quality of primary studies of measurement properties, and to describe how authors evaluate results of the studies. Literature searches were performed in three databases. One hundred and forty-eight reviews were included. The purpose of included reviews was to identify health status instruments used in an evaluative application and to report on the measurement properties of these instruments. Two independent reviewers selected the articles and extracted the data. Reviews were often of low quality: 22% of the reviews used one database, the search strategy was often poorly described, and in many cases it was not reported whether article selection (75%) and data extraction (71%) was done by two independent reviewers. In 11 reviews the methodological quality of the primary studies was evaluated for all measurement properties, and of these 11 reviews only 7 evaluated the results. Methods to evaluate the quality of the primary studies and the results differed widely. The poor quality of reviews hampers evidence-based selection of instruments. Guidelines for conducting and reporting systematic reviews of measurement properties should be developed.
Overview
CoChat Summary
TL;DR
This paper appraises 148 systematic reviews of health status measurement instruments and finds that their methodological quality was often poor, especially for search reporting and the assessment of primary studies. The authors conclude that guidelines for conducting and reporting systematic reviews of measurement properties are needed.
Key findings
- The search identified 7,779 records, of which 148 systematic reviews of measurement properties were included.
- In 84% of reviews, the search strategy was described, but it was often limited; 22% used only one database. The databases most often used were PubMed (93%), PsycINFO (40%), CINAHL (39%), EMBASE (35%), and the Cochrane library (16%).
- In 44% (65/148) of reviews, the methodological quality of the included studies was not assessed and the results were not appraised; in 56% (83/148), quality was (partly) assessed and some results were evaluated.
- Among the 83 reviews that assessed quality/results, 53% (44/83) applied both standards and criteria of adequacy, 46% (38/83) applied only criteria of adequacy, and 1 review applied only standards. Only 11 reviews described and applied a complete set of fully reproducible standards, and of these 11, only 7 evaluated the results.
Methods
- The authors searched PubMed (to March 2007), EMBASE (to March 2007), and PsycINFO (to June 2005), plus reference lists and authors’ own literature, to identify systematic reviews of measurement properties of health status instruments.
- Included reviews had to claim to be systematic reviews, aim to identify all available health status measurement instruments for a specified population, concern instruments used in an evaluative/longitudinal context, and report on or evaluate measurement properties.
- Two reviewers independently selected articles and extracted data on review characteristics, review-process quality, assessment of primary-study methodological quality, evaluation of results, and whether an overall conclusion was synthesized.
Limitations
- The paper states that, because of the considerable variation in design, performance, and data presentation of the included reviews, some degree of judgement was unavoidable in appraising quality and describing standards.
- The authors also note that they only used information from the published reviews and did not contact authors for additional information, so they may have slightly underrated review quality.
- Evident from the text: the review was limited to English-language reviews and to reviews that performed at least one electronic database search, so relevant non-English or non-electronically searched reviews were excluded.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 2 Not valid · 14 Not checked · 61 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 11 reviews the methodological quality of the primary studies was evaluated for all measurement properties, and of these 11 reviews only 7 evaluated the results.”
- Also stated in: § J. Alonso · plain
“In 11 reviews the methodological quality of the primary studies was evaluated for all measurement properties”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § J. Alonso · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Publication of systematic reviews of measurement properties has increased from less than one review per year in the 1990s up to 31 in 2005 (Fig. 2).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“in many cases it was not reported whether article selection (75%) and data extraction (71%) was done by two independent reviewers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 84% of the reviews the authors described the search strategy in some way.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“22% of the reviews used one database”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
41 more claims
Create a free account to see all 80 claims and to verify the 34 references still open.
- Probably not a substantial claim. 21
- Borderline: one of the tests for a substantial claim is unclear. 1
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Psychometric systematic reviews (or systematic reviews of measurement properties) are conducted to assess the quality/characteristics of health measurement instruments to determine the best tool for use (in terms of its validity, reliability, responsiveness etc.) in practice for a certain condition or factor [31–33].”
From What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences (2018) · cites this paper as [33]
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · cited 3×
“These guidelines contain items on the quality of the search strategy [4], article selection and data extraction [3, 7, 8], and inclusion and exclusion criteria [6].”
BackgroundMethodQuality criteria were proposed for measurement properties of health status questionnaires
Terwee, C. B., Bot, S. D. M., de Boer, M. R., van der Windt, D. A. W. M., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. W. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.012 · Cited by 10,764 · cited 1×
“Seven out of these 11 studies used Terwee criteria [7] for the quality assessment.”
MethodSelf-citationDevelopment of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Self-citation(2000) · Cited by 4,106
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Wilson, I. B. (1995) · JAMA · doi:10.1001/jama.1995.03520250075037 · Cited by 3,027 · cited 2×
“We considered the following concepts to represent ‘‘health status’’ based on the model of Wilson and Cleary [14]: biological and physiological processes, symptoms, functional status (i.e., both physical functioning and psychosocial functioning), or general health perceptions.”
MethodU.S. Department of Health and Human Services FDA Center for Drug Evaluation and Research, U.S. Department of Health and Human Services FDA Center for Biologics Evaluation and Research, & U.S. Department of Health and Human Services FDA Center for Devices and Radiological Health (2006) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-4-79 · Cited by 2,856
Verhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., Bouter, L. M., & Knipschild, P. G. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00131-0 · Cited by 2,379 · cited 1×
“, Delphi List [9]) or diagnostic studies (QUADAS [10]).”
MethodSelf-citationAssessing health status and quality-of-life instruments: Attributes and review criteria
Lohr, K. N. (2002) · Quality of Life Research · doi:10.1023/a:1015291021312 · Cited by 2,269
Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group
van Tulder, M., Furlan, A., Bombardier, C., & Bouter, L. (2003) · Spine · doi:10.1097/01.brs.0000065484.95996.af · Cited by 2,180 · cited 2×
“These guidelines contain items on the quality of the search strategy [4], article selection and data extraction [3, 7, 8], and inclusion and exclusion criteria [6].”
BackgroundSelf-citation
Research works printed but not linked: 1
- Higgins, J. P. T., & Green, S. (Eds.). (2006). Cochrane handbook for systematic reviews of interventions. Version 5.0.1 (update September 2008). The Cochrane Collaboration, 2008. Available from www.cochrane-handbook.org .
Cited by
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465
“The Steering Committee systematically searched the literature to find systematic reviews on measurement properties of health status measurement instruments [1] and methodological articles and textbooks that provided terminology and definitions of measurement properties.”
BackgroundSelf-citationMokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9606-8 · Cited by 4,260
“For each measurement property, we found several different standards, some of which were contradictory [9].”
BackgroundResultSelf-citationCOSMIN guideline for systematic reviews of patient-reported outcome measures
Prinsen, C. A. C., Mokkink, L. B., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1798-3 · Cited by 3,345
“In addition, we have studied the quality of systematic reviews of PROMs in two consecutive reviews [7, 17], and in reviews that have used the COSMIN methodology we have specifically searched for the comments made by review authors relating to the COSMIN methodology.”
MethodSelf-citationPilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., Cella, D., & PROMIS Cooperative Group (2011) · Assessment · doi:10.1177/1073191111411667 · Cited by 2,132
“An important component of content validity is the extent to which items reflect the perspective of the population of interest (Brod, Tesler, & Christensen, 2009; Food and Drug Administration, 2009; Mokkink et al., 2009).”
BackgroundTerwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W. J. G., Bouter, L. M., & de Vet, H. C. W. (2012) · Quality of Life Research · doi:10.1007/s11136-011-9960-1 · Cited by 1,978
“Systematic reviews of measurement properties are useful for selecting the best measurement instrument for a specific purpose [1].”
BackgroundSelf-citation
Metadata
- Authors
- Mokkink, L. B., Terwee, C. B., Stratford, P. W., Alonso, J., Patrick, D. L., Riphagen, I., Knol, D. L., Bouter, L. M., & de Vet, H. C. W.
- Affiliations
- Amsterdam Public Health (NL)
- McMaster University (CA)
- Municipal Institute for Medical Research (ES)
- University of Washington (US)
- Norwegian University of Science and Technology (NO)
- Corresponding author
- Mokkink, Lidwine B.
- ORCID
- Venue
- Quality of Life Research
- Published
- 2009-02-24
- Volume
- 18
- Issue
- 3
- Pages
- 313-333
- ISSN
- 0962-9343 (ISSN-L)
- 1573-2649
- OpenAlex
- W1989226082
- Cited by
- 186
- References
- 178
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 17 · 2023: 9 · 2022: 13 · 2021: 5 · 2020: 10 · 2019: 10 · 2018: 24 · 2017: 21 · 2016: 16 · 2015: 11 · 2014: 12 · 2013: 7 · 2012: 8
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- creativecommons.org/licenses/by-nc/2.0 (tdm)
- Locations
- Quality of Life Research (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- VU Research Portal (publishedVersion, other-oa)
- Pure Amsterdam UMC (submittedVersion)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (submittedVersion)
- Digital Academic REpository of VU University Amsterdam (Vrije Universiteit Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Vrije Universiteit Amsterdam
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- methodological quality, measurement properties, reporting quality, search strategies, inter-rater reliability, instrument selection, reporting guidelines
- MeSH terms
- Adolescent, Adult, Child, Child, Preschool, Evaluation Studies as Topic, Female, Health Status, Humans, Infant, Male, Middle Aged, Quality of Life, Surveys and Questionnaires, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation of the methodological quality of systematic reviews of health status measurement instruments. https://app.cochat.ai/papers/10.1007/s11136-009-9451-9
- The paper
- Mokkink, L. B., Terwee, C. B., Stratford, P. W., Alonso, J., Patrick, D. L., Riphagen, I., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2009). Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research. https://doi.org/10.1007/s11136-009-9451-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.