CoChat
Sign Up
Cited by 90Meta-analysis and systematic reviewsHealth Education and ValidationHealth Sciences Research and Education

The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT)

Open access
Rosella, L., Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V.·2016·Public Health
DOI 10.1016/j.puhe.2015.10.027
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVES: Most quality appraisal tools were developed for clinical medicine and tend to be study-specific with a strong emphasis on risk of bias. In order to be more relevant to public health, an appropriate quality appraisal tool needs to be less reliant on the evidence hierarchy and consider practice applicability. Given the broad range of study designs used in public health, the objective of this study was to develop and validate a meta-tool that combines public health-focused principles of appraisal coupled with a set of design-specific companion tools. STUDY DESIGN: Several design methods were used to develop and validate the tool including literature review, synthesis, and validation with a reference standard. METHODS: A search of critical appraisal tools relevant to public health was conducted; core concepts were collated. The resulting framework was piloted during three feedback sessions with public health practitioners. Following subsequent revisions, the final meta-tool, the Meta Quality Appraisal Tool (MetaQAT), was then validated through a content analysis of appraisals conducted by two groups of experienced public health researchers (MetaQAT vs generic appraisal form). RESULTS: The MetaQAT framework consists of four domains: relevancy, reliability, validity, and applicability. In addition, a companion tool was assembled from existing critical appraisal tools to provide study design-specific guidance on validity appraisal. Content analysis showed similar methodological and generalizability concerns were raised by both groups; however, the MetaQAT appraisers commented more extensively on applicability to public health practice. CONCLUSIONS: Critical appraisal tools designed for clinical medicine have limitations for use in the context of public health. The meta-tool structure of the MetaQAT allows for rigorous appraisal, while allowing users to simultaneously appraise the multitude of study designs relevant to public health research and assess non-standard domains, such as applicability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.puhe.2015.10.027

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • All authors codeveloped a structured risk-of-bias and quality assessment based on the MetaQAT framework to evaluate relevance, reliability, applicability, and validity.

    “Because included studies used diverse observational designs, all authors codeveloped a structured risk-of-bias and quality assessment based on the MetaQAT framework to evaluate relevance, reliability, applicability, and validity.13”

    From Systematic Review and Meta-Analyses of Time-Use, Productivity, and Labor Outcomes Among Unpaid Cancer Caregivers (2026) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Heller RF, Verma A, Gemmell I, Harrison R, Hart J, Edwards R. Critical appraisal for public health: a new checklist. Public Health 2008;122(1):92e8.
  2. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health 2009;30:175e201.
  3. Petticrew M, Roberts H. Evidence, hierarchies, and typologies: horses for courses. J Epidemiol Community Health(7):527e9 [Internet]. [cited 2015 Feb 10]. Available from: http://jech.bmj. com/content/57/7/527.full.pdfþhtml, 2003;57.
  4. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol:22 [Internet]. [cited 2014 Aug 7]. Available from: http://www.biomedcentral.com/ content/pdf/1471-2288-4-22.pdf, 2004;4.
  5. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol:10 [Internet]. [cited 2014 Aug 1]. Available from: http://www.biomedcentral.com/content/pdf/ 1471-2288-7-10.pdf, 2007;7.
  6. Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health 2013;67(1):98e104.
  7. Wada K, Oka-Ezoe K, Smith DR. Wearing face masks in public during the influenza season may reflect other positive hygiene practices in Japan. BMC Public Health:1065 [Internet]. [cited 2013 May 1]. Available from: http://www. biomedcentral.com/content/pdf/1471-2458-12-1065.pdf, 2012;12.
  8. Singleton RA, Straits BC. Approaches to social research. New York: Oxford University Press; 2010115e49. Chapter 5, Measurement.
  9. Cocoros NM, Lash TL, Norgaard M, Farkas DK, DeMaria A, Sorensen HT. Hospitalized prenatal and childhood infections and obesity in Danish male conscripts. Ann Epidemiol 2013;23(6):307e13.
  10. Ojha RP, Tota JE, Orrutt-Powell TN, Klosky JL, Ashokkumar R, Gurney JG. The accuracy of human papillomavirus vaccination status based on adult proxy recall or household immunization records for adolescent females in the United States: results from the National Immunization Survey-Teen. Ann Epidemiol 2013;23(5):281e5.
  11. Selassie AW, Wilson DA, Pickelsimer EE, Voronca DC, Williams NR, Edwards JC. Incidence of sport-related traumatic brain injury and risk factors of severity: a population-based epidemiologic study. Ann Epidemiol 2013;23(12):750e6. Available from:.
  12. Booth A, Brice A. Column editors. HILJ. 2001;18(3):175e77.
  13. Centre for Evidence-Based Medicine. Critical appraisal worksheets: systematic review critical appraisal sheet [Internet]. Oxford: Centre for Evidence-Based Medicine [cited 2015 Feb 10]. Available from:, http://www.cebm.net/critical-appraisal/; 2014.
  14. Centre for Evidence-Based Medicine. Critical appraisal worksheets: therapy/RCT critical appraisal sheet [Internet]. Oxford: Centre for Evidence-Based Medicine [cited 2015 Feb 10]. Available from:, http://www.cebm.net/critical-appraisal/; 2014.
  15. Critical Appraisal Skills Programme (CASP). CASP: case control studies [Internet]. Oxford: Critical Appraisal Skills Programme [cited 2015 Feb 10]. Available from:, http://www.casp-uk.net/; 2010.
  16. Critical Appraisal Skills Programme (CASP). CASP: cohort study checklist [Internet]. Oxford: CASP [cited 2015 Feb 10]. Available from:, http://media.wix.com/ugd/dded87_ 36c5c76519f7bf14731ed1985e8e9798.pdf; 2013.
  17. Critical Appraisal Skills Programme (CASP). CASP: economic evaluations checklist [Internet]. Oxford: CASP [cited 2015 Feb 10]. Available from:, http://media.wix.com/ugd/dded87_ ff37ab65474ac19e19ec31d5f2e4887d.pdf; 2013.
  18. Critical Appraisal Skills Programme (CASP). CASP: qualitative research checklist [Internet]. Oxford: CASP [cited 2015 Feb 10]. Available from:, http://media.wix.com/ugd/dded87_ 951541699e9edc71ce66c9bac4734c69.pdf; 2013.
  19. Critical Appraisal Skills Programme (CASP). CASP: randomized controlled trials checklist [Internet]. Oxford: CASP [cited 2015 Feb 10]. Available from:, http://media.wix.com/ugd/dded87_ c7984187fdc74d25a3fba6c707383b15.pdf; 2013.
  20. Critical Appraisal Skills Programme (CASP). CASP: systematic reviews checklist [Internet]. Oxford: CASP [cited 2015 Feb 10]. Available from:, http://media.wix.com/ugd/dded87_ 37491d0241aa448a8f3d4ae17c869472.pdf; 2013.
  21. Grimmer K, Dizon JM, Milanese S, King E, Beaton K, Thorpe O, et al. Efficient clinical evaluation of guideline quality: development and testing of a new tool. BMC Med Res Methodol:63 [Internet]. [cited 2014 Dec 1]. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-14- 63.pdf, 2014;14.
  22. Health Evidence. Quality assessment tool e review articles [Internet]. Hamilton, ON: Health Evidence [cited 2015 Feb 10]. Available from:, http://www.healthevidence.org/documents/ our-appraisal-tools/QA_tool&dictionary_18.Mar.2013.pdf; 2013.
  23. Jackson R, Ameratunga S, Broad J, Connor J, Lethaby A, Robb G, et al. The GATE frame: critical appraisal with pictures. Evid Based Med 2006;11(2):35e8.
  24. Centre for Evidence-Based Medicine. Systematic review (of therapy) critical appraisal worksheet [Internet]. Ottawa, ON: KT Clearinghouse [cited 2015 Feb 10]. Available from:, http:// ktclearinghouse.ca/cebm/teaching/worksheets/sr; 2014.
  25. Law M, Stewart D, Pollock N, Letts L, Bosch J, Westmorland M. Critical review form e quantitative studies [Internet]. Hamilton, ON: McMaster University [cited 2015 Feb 10]. Available from:, http://www.srs-mcmaster.ca/Portals/20/pdf/ebp/quanreview. pdf; 1998.
  26. Letts L, Wilkins S, Law M, Steward D, Bosch J, Westmorland M. Critical review form e qualitative studies. Version 2.0 [Internet]. Hamilton, ON: McMaster University [cited 2015 Feb 10]. Available from:, http://www.srs- mcmaster.ca/Portals/20/pdf/ebp/qualreview_version2.0.pdf; 2007.
  27. MacAuley D. READER: an acronym to aid critical reading by general practitioners. Br J Gen Pract(379):83e5 [Internet]. [cited 2015 Feb 10]. Available from: http://www.ncbi.nlm.nih.gov/pmc/ articles/PMC1238789/pdf/brjgenprac00035-0039.pdf, 1994;44.
  28. Moher D, Liberati A, Tetzlaff J, Atlman DGPRISMA Group. Preferred reporting items for systematic reviews and meta- analyses: the PRISMA statement. BMJ:b2535 [Internet]. [cited 2015 Feb 10]. Available from: http://www.bmj.com/content/ 339/bmj.b2535.long, 2009;339.
  29. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care(1):18e23 [Internet]. [cited 2015 Feb 10]. Available from: http://qualitysafety.bmj.com/content/12/1/18.full. pdfþhtml, 2003;12.
  30. Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta- analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 2006;59(12):1249e56.
  31. Wendt O, Miller B. Quality appraisal of single-subject experimental designs: an overview and comparison of different appraisal tools. Educ Treat Child 2012;35(2):235e68.
  32. Harder T, Takla A, Rehfuess E, Sanchez-Vivar A, Mtysiak- Klose D, Echmanns T, et al. Evidence-based decision-making in infectious diseases epidemiology, prevention and control: matching research questions to study designs and quality appraisal tools. BMC Med Res Methodol:69 [Internet]. [cited 2014 Dec 1]. Available from: http://www.biomedcentral.com/ content/pdf/1471-2288-14-69.pdf, 2014;14.

Cited by

Sorted by citations

Authors
Rosella, L., Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., & Goel, V.
Affiliations
  • University of Toronto (CA)
  • Public Health Ontario (CA)
Corresponding author
Rosella, L.
ORCID
Venue
Public Health
Publisher
Elsevier BV
Published
2016-07
Volume
136
Pages
57-65
ISSN
0033-3506 (ISSN-L)
DOI
10.1016/j.puhe.2015.10.027
OpenAlex
W2300742922
Cited by
90
References
8
Citations by year
2026: 6 · 2025: 12 · 2024: 19 · 2023: 9 · 2022: 10 · 2021: 12 · 2020: 6 · 2019: 7 · 2018: 5 · 2017: 2 · 2016: 1 · 2015: 1
FWCI
4.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Public Health (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Education › Health Education and Validation
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
critical appraisal tools, quality assessment, public health evidence, applicability, validity, reliability statistics, evidence hierarchy, tool validation, public health research
MeSH terms
Humans, Pilot Projects, Public Health, Research Design, Meta-Analysis as Topic, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review