Cited by 1,479Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyDelphi Technique in Research

The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index; none of the 5 sources its claims cite is checked for support yet.

The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence

Open access
Munn, Zachary, Moola, Sandeep, Riitano, Dagmara, Lisy, Karolina·2014·International Journal of Health Policy and Management

Abstract

BACKGROUND: Recently there has been a significant increase in the number of systematic reviews addressing questions of prevalence. Key features of a systematic review include the creation of an a priori protocol, clear inclusion criteria, a structured and systematic search process, critical appraisal of studies, and a formal process of data extraction followed by methods to synthesize, or combine, this data. Currently there exists no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data. METHODS: A working group was created to assess current critical appraisal tools for studies reporting prevalence data and develop a new tool for these studies in systematic reviews of prevalence. Following the development of this tool it was piloted amongst an experienced group of sixteen healthcare researchers. RESULTS: The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews. Participants found the tool acceptable and easy to use. Some comments were provided which helped refine the criteria. CONCLUSION: The results of this pilot study found that this tool was well-accepted by users and further refinements have been made to the tool based on their feedback. We now put forward this tool for use by authors conducting prevalence systematic reviews.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper describes development and pilot testing of a 10-item critical appraisal tool for studies reporting prevalence data in systematic reviews. In a pilot with 16 experienced healthcare researchers, the tool was rated acceptable, easy to use, timely, and valid by all but one participant, though further psychometric testing was recommended.

Key findings

  • The working group found no standard method for critically appraising prevalence studies and developed a tool addressing representative sampling, recruitment, sample size, description of subjects/setting, data coverage, objective measurement, reliability, statistical analysis, confounding, and subpopulation criteria.
  • Among 16 pilot participants, mean ease of use was 3.63 on a 5-point scale, with 75% rating it 4 (“easy”); mean acceptability was 4.33, with all participants rating 4 or 5; mean timeliness was 3.94, with 88% rating 4 (“acceptable”).
  • All participants except 1 viewed the tool as a valid quality appraisal checklist for prevalence data.
  • Feedback led to changes in question order and supporting information, but no changes were made to the individual questions; the tool was to be incorporated into the next Joanna Briggs Institute systematic review package.

Methods

  • A Joanna Briggs Institute working group searched for systematic reviews of prevalence data and existing critical appraisal tools, then developed a new tool based on identified criteria and their own knowledge/research; the tool was internally trialed and refined.
  • The tool was piloted at the 2013 Joanna Briggs Institute convention in Adelaide in October; attendees appraised a cross-sectional study using the new tool and completed a short survey on face validity, ease of use, acceptability, timeliness, and areas for improvement.
  • Sixteen workshop participants completed the task and survey; 13 had an academic/research background, 2 had a health background, and 1 had both, with average research experience of 11 years (range 2.5–30 years).

Limitations

  • The paper states the tool needs further testing in a larger-scale study to assess other clinimetric properties, particularly construct validity and inter-rater reliability.
  • Evident from the text: the pilot assessed only face validity, ease of use, acceptability, and timeliness, with no formal inter-rater reliability or construct validity testing.
  • Evident from the text: the pilot used a small convenience sample of 16 workshop attendees, mostly with academic/research backgrounds, which may limit generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.15171/ijhpm.2014.71

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 5 cited sources checked · 0 supported · 0 flagged

Show all 23 claims

  • Probably not a substantial claim. 11
  • It may state two separate claims as one. 4
  • The quote only partly states this claim. 6
14 of 14
  • The paper relies on this without support.

    Policy-makers are often faced with conflicting reports of disease prevalence in the literature.

    “However, policy-makers are often faced with conflicting reports of disease prevalence in the literature.”
  • The paper relies on this without support.

    User acceptance and face validity of the tool indicate its validity as a quality appraisal checklist.

    “The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews.”
    “along with a short survey that was developed to establish the face validity, ease of use, acceptability and timeliness (i.e.time taken to complete) of the tool, and feedback on areas for improvement (23).”
  • The paper relies on this without support.

    The working group's search for critical appraisal tools was comprehensive enough to conclude that no standard method exists.

    “Currently there exists no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data.”
    “The group then searched for critical appraisal tools that have been used to assess studies reporting on prevalence data.”
  • The paper relies on this without support.

    The tool's criteria cover all important aspects of methodological quality for prevalence studies.

    “The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews.”
    “Based on a review of these criteria and our own knowledge and research we developed a tool specifically for use in systematic reviews of prevalence data.”
  • The paper relies on this without support.

    The working group's search for systematic reviews of prevalence was comprehensive enough to conclude that no formal guidance exists.

    “However, currently there does not appear to be any formal guidance for authors wishing to conduct a review of prevalence.”
    “The working party began by conducting a search for systematic reviews of prevalence data to determine how the methodological quality of studies included in these reviews were assessed.”
  • The paper relies on this without support.

    There has been substantial variability in how studies reporting prevalence data are critically appraised.

    “Until now there has been substantial variability in how studies reporting prevalence data are critically appraised.”
  • The paper relies on this without support.

    Systematic reviews of prevalence and incidence data are becoming increasingly important and popular.

    “Systematic reviews of prevalence and incidence data are becoming increasingly important as decision makers realize their usefulness in informing policy and practice.”
    “Prevalence systematic reviews are becoming increasingly popular within the research community.”
  • The paper relies on this without support.

    Many health problems are not easily diagnosed or defined, and some measures may not be capable of including or excluding appropriate levels or stages of the health problem.

    “Many health problems are not easily diagnosed or defined and some measures may not be capable of including or excluding appropriate levels or stages of the health problem.”
  • The paper relies on this without support.

    If outcomes are assessed using observer-reported or self-reported scales, the risk of over- or under-reporting is increased, and objectivity is compromised.

    “If the outcomes were assessed using observer reported, or self-reported scales, the risk of over-or under-reporting is increased, and objectivity is compromised.”
  • The paper relies on this without support.

    Validated measurement tools have a significant impact on outcome assessment validity.

    “Importantly, determine if the measurement tools used were validated instruments as this has a significant impact on outcome assessment validity.”
  • The paper relies on this without support.

    Considerable judgment is required to determine the presence of some health outcomes.

    “Considerable judgment is required to determine the presence of some health outcomes.”
  • The paper relies on this without support.

    Differing methods of analysis are based on differing assumptions about the data and how it will respond.

    “differing methods of analysis are based on differing assumptions about the data and how it will respond.”
  • The paper relies on this without support.

    Prevalence rates found in studies only provide estimates of the true prevalence of a problem in the larger population.

    “Prevalence rates found in studies only provide estimates of the true prevalence of a problem in the larger population.”
  • The paper relies on this without support.

    Since some subgroups are very small, 95% confidence intervals are usually given.

    “Since some subgroups are very small, 95% confidence intervals are usually given.”
  • The paper relies on this without support.

    Incidence and prevalence studies often draw or report findings regarding the differences between groups.

    “Incidence and prevalence studies often draw or report findings regarding the differences between groups.”
15 of 15

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 23 references

Research works printed but not linked: 2

  1. The Joanna Briggs Institute. Joanna Briggs Institute Reviewers' Manual: 2011 edition. Adelaide: The Joanna Briggs Institute; 2011.
  2. Munn Z, Moola S, Lisy K, Riitano D. The Systematic Review of Prevalence and Incidence Data The Joanna Briggs Institute Reviewer's Manual 2014. Australia: The Joanna Briggs Institute; 2014.

Cited by

Sorted by citations

Items 501–600 of 999

Metadata

Authors
Munn, Z., Moola, S., Riitano, D., & Lisy, K.
Affiliations
The University of Adelaide (AU)
Venue
International Journal of Health Policy and Management
Published
2014
Volume
3
Issue
3
Pages
123-128
ISSN
2322-5939 (ISSN-L)
OpenAlex
W2167191881
Cited by
1,479
References
23
Citations by year
2026: 145 · 2025: 166 · 2024: 177 · 2023: 193 · 2022: 220 · 2021: 172 · 2020: 151 · 2019: 93 · 2018: 73 · 2017: 37 · 2016: 39 · 2015: 11 · 2014: 2
FWCI
8.12
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • International Journal of Health Policy and Management (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Research Information System of Ardabil University of Medical Sciences (Ardabil University of Medical Sciences) (acceptedVersion)
  • SSRN Electronic Journal (submittedVersion)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
critical appraisal tools, prevalence, systematic reviews, methodological quality assessment, prevalence data, tool development, pilot study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review