CoChat
Sign Up
Cited by 1,479Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyDelphi Technique in Research

The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence

Open access
Munn, Zachary, Moola, Sandeep, Riitano, Dagmara, Lisy, Karolina·2014·International Journal of Health Policy and Management
DOI 10.15171/ijhpm.2014.71PMC PMC4154549
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Recently there has been a significant increase in the number of systematic reviews addressing questions of prevalence. Key features of a systematic review include the creation of an a priori protocol, clear inclusion criteria, a structured and systematic search process, critical appraisal of studies, and a formal process of data extraction followed by methods to synthesize, or combine, this data. Currently there exists no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data. METHODS: A working group was created to assess current critical appraisal tools for studies reporting prevalence data and develop a new tool for these studies in systematic reviews of prevalence. Following the development of this tool it was piloted amongst an experienced group of sixteen healthcare researchers. RESULTS: The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews. Participants found the tool acceptable and easy to use. Some comments were provided which helped refine the criteria. CONCLUSION: The results of this pilot study found that this tool was well-accepted by users and further refinements have been made to the tool based on their feedback. We now put forward this tool for use by authors conducting prevalence systematic reviews.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper describes development and pilot testing of a 10-item critical appraisal tool for studies reporting prevalence data in systematic reviews. In a pilot with 16 experienced healthcare researchers, the tool was rated acceptable, easy to use, timely, and valid by all but one participant, though further psychometric testing was recommended.

Key findings

  • The working group found no standard method for critically appraising prevalence studies and developed a tool addressing representative sampling, recruitment, sample size, description of subjects/setting, data coverage, objective measurement, reliability, statistical analysis, confounding, and subpopulation criteria.
  • Among 16 pilot participants, mean ease of use was 3.63 on a 5-point scale, with 75% rating it 4 (“easy”); mean acceptability was 4.33, with all participants rating 4 or 5; mean timeliness was 3.94, with 88% rating 4 (“acceptable”).
  • All participants except 1 viewed the tool as a valid quality appraisal checklist for prevalence data.
  • Feedback led to changes in question order and supporting information, but no changes were made to the individual questions; the tool was to be incorporated into the next Joanna Briggs Institute systematic review package.

Methods

  • A Joanna Briggs Institute working group searched for systematic reviews of prevalence data and existing critical appraisal tools, then developed a new tool based on identified criteria and their own knowledge/research; the tool was internally trialed and refined.
  • The tool was piloted at the 2013 Joanna Briggs Institute convention in Adelaide in October; attendees appraised a cross-sectional study using the new tool and completed a short survey on face validity, ease of use, acceptability, timeliness, and areas for improvement.
  • Sixteen workshop participants completed the task and survey; 13 had an academic/research background, 2 had a health background, and 1 had both, with average research experience of 11 years (range 2.5–30 years).

Limitations

  • The paper states the tool needs further testing in a larger-scale study to assess other clinimetric properties, particularly construct validity and inter-rater reliability.
  • Evident from the text: the pilot assessed only face validity, ease of use, acceptability, and timeliness, with no formal inter-rater reliability or construct validity testing.
  • Evident from the text: the pilot used a small convenience sample of 16 workshop attendees, mostly with academic/research backgrounds, which may limit generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.15171/ijhpm.2014.71

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 5 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The major area where prevalence reviews were disparate was in their conduct of critical appraisal or quality assessment of included studies.

    “The group identified that the major area where prevalence reviews were disparate was in their conduct of critical appraisal or quality assessment of included studies.”
    Quote in the paperSubstantial claim
  • Findingown data

    The new tool is simple.

    “Our tool also has the advantage of being simple, easy and quick as shown during the pilot testing.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    The current checklists identified from the authors' search did not sufficiently address important quality issues in prevalence studies.

    “We have developed this tool as we did not feel that any of the current checklists identified from our search sufficiently addressed important quality issues in prevalence studies.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the pilot test, the tool had timeliness to complete.

    “A pilot test of this tool amongst a group of experienced healthcare professionals and researchers found that this tool had face validity and high levels of acceptability, ease of use and timeliness to complete.”
    • Also stated in: § Discussion · plain
      “Our tool also has the advantage of being simple, easy and quick as shown during the pilot testing.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The tool was easy to use for pilot participants.

    “Participants found the tool acceptable and easy to use.”
    • Also stated in: § 10. · plain
      “participants providing a rating of 4, corresponding to 'easy' .”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 11
  • May combine two claims. 4
  • The quote only partly states this claim. 6

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The average time spent working in research among participants was 11 years, with a minimum of 2.5 years and a maximum of 30 years.

    “The average time spent working in research was 11 years, with the minimum being 2.5 years and the maximum experience being 30 years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The refinements consisted of changes in the order of questions and supporting information, with no changes to individual questions.

    “These comments resulted in some changes in the order of the questions of the tool and the supporting information used to assist in judging criteria, although no changes were made to the individual questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The tool was timely to complete for pilot participants.

    “For timeliness, the mean score was 3.94, with 88% providing a score of 4 (acceptable).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Of the sixteen participants, 13 had an academic or research background, 2 had a health background, and one had both.

    “Of the 16, 13 participants stated they had an academic/research background, 2 said they had a health background, and one said they had both.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Currently there is no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data.

    “Currently there exists no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The working group's search for critical appraisal tools was comprehensive enough to conclude that no standard method exists.

  • The quote only partly states this claim.

    Findingown data

    The mean acceptability was 4.33 (95% CI 4.06 to 4.60) on a 5-point Likert scale.

    “| Arithmetic mean | 11.37 | 3.63 | 4.33 | 3.94 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% lower confidence limit | 6.94 | 3.24 | 4.06 | 3.63 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% upper confidence limit | 15.80 | 4.00 | 4.60 | 4.24 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The mean time was 3.94 (95% CI 3.63 to 4.24) on a 5-point Likert scale.

    “| Arithmetic mean | 11.37 | 3.63 | 4.33 | 3.94 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% lower confidence limit | 6.94 | 3.24 | 4.06 | 3.63 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% upper confidence limit | 15.80 | 4.00 | 4.60 | 4.24 |”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The mean ease of tool use was 3.63 (95% CI 3.24 to 4.00) on a 5-point Likert scale.

    “| Arithmetic mean | 11.37 | 3.63 | 4.33 | 3.94 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% lower confidence limit | 6.94 | 3.24 | 4.06 | 3.63 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% upper confidence limit | 15.80 | 4.00 | 4.60 | 4.24 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The mean experience was 11.37 (95% CI 6.94 to 15.80).

    “| Arithmetic mean | 11.37 | 3.63 | 4.33 | 3.94 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% lower confidence limit | 6.94 | 3.24 | 4.06 | 3.63 |”
    • Also stated in: § 10. Were subpopulations identified using objective criteria? · plain
      “| 95.00% upper confidence limit | 15.80 | 4.00 | 4.60 | 4.24 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Findingnone given

    The survey pilot included 16 cases for ease of tool use and time.

    “| Number of cases | 15 | 16 | 15 | 16 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The survey pilot included 15 cases for experience and acceptability.

    “| Number of cases | 15 | 16 | 15 | 16 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Sixteen workshop participants completed the critical appraisal task and survey.

    “Sixteen workshop participants completed the critical appraisal task and survey.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The systematic review is well-accepted as the ideal method to summarize literature relating to a social or healthcare topic.

    “The systematic review of evidence has been proposed and is now well-accepted as the ideal method to summarize the literature relating to a certain social or healthcare topic (2,3).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [2]

      Not checked in the free run.

    • cites [3]

      Not checked in the free run.

    • Rests on an assumption: Policy-makers are often faced with conflicting reports of disease prevalence in the literature.

  • May combine two claims.

    Premisereasoning

    The Hoy et al. tool contains similar questions to the new tool but there are important differences.

    “Some of the tools [most notably the tool refined by Hoy et al. (20)] contain similar questions to our tool but there are important differences.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20]

      Not checked in the free run.

14 of 14

  • The paper relies on this without support.

    Policy-makers are often faced with conflicting reports of disease prevalence in the literature.

    “However, policy-makers are often faced with conflicting reports of disease prevalence in the literature.”

  • The paper relies on this without support.

    User acceptance and face validity of the tool indicate its validity as a quality appraisal checklist.

    “The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews.”
    “along with a short survey that was developed to establish the face validity, ease of use, acceptability and timeliness (i.e.time taken to complete) of the tool, and feedback on areas for improvement (23).”

  • The paper relies on this without support.

    The working group's search for critical appraisal tools was comprehensive enough to conclude that no standard method exists.

    “Currently there exists no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data.”
    “The group then searched for critical appraisal tools that have been used to assess studies reporting on prevalence data.”

  • The paper relies on this without support.

    The tool's criteria cover all important aspects of methodological quality for prevalence studies.

    “The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews.”
    “Based on a review of these criteria and our own knowledge and research we developed a tool specifically for use in systematic reviews of prevalence data.”

  • The paper relies on this without support.

    The working group's search for systematic reviews of prevalence was comprehensive enough to conclude that no formal guidance exists.

    “However, currently there does not appear to be any formal guidance for authors wishing to conduct a review of prevalence.”
    “The working party began by conducting a search for systematic reviews of prevalence data to determine how the methodological quality of studies included in these reviews were assessed.”

  • The paper relies on this without support.

    There has been substantial variability in how studies reporting prevalence data are critically appraised.

    “Until now there has been substantial variability in how studies reporting prevalence data are critically appraised.”

  • The paper relies on this without support.

    Systematic reviews of prevalence and incidence data are becoming increasingly important and popular.

    “Systematic reviews of prevalence and incidence data are becoming increasingly important as decision makers realize their usefulness in informing policy and practice.”
    “Prevalence systematic reviews are becoming increasingly popular within the research community.”

  • The paper relies on this without support.

    Many health problems are not easily diagnosed or defined, and some measures may not be capable of including or excluding appropriate levels or stages of the health problem.

    “Many health problems are not easily diagnosed or defined and some measures may not be capable of including or excluding appropriate levels or stages of the health problem.”

  • The paper relies on this without support.

    If outcomes are assessed using observer-reported or self-reported scales, the risk of over- or under-reporting is increased, and objectivity is compromised.

    “If the outcomes were assessed using observer reported, or self-reported scales, the risk of over-or under-reporting is increased, and objectivity is compromised.”

  • The paper relies on this without support.

    Validated measurement tools have a significant impact on outcome assessment validity.

    “Importantly, determine if the measurement tools used were validated instruments as this has a significant impact on outcome assessment validity.”

  • The paper relies on this without support.

    Considerable judgment is required to determine the presence of some health outcomes.

    “Considerable judgment is required to determine the presence of some health outcomes.”

  • The paper relies on this without support.

    Differing methods of analysis are based on differing assumptions about the data and how it will respond.

    “differing methods of analysis are based on differing assumptions about the data and how it will respond.”

  • The paper relies on this without support.

    Prevalence rates found in studies only provide estimates of the true prevalence of a problem in the larger population.

    “Prevalence rates found in studies only provide estimates of the true prevalence of a problem in the larger population.”

  • The paper relies on this without support.

    Since some subgroups are very small, 95% confidence intervals are usually given.

    “Since some subgroups are very small, 95% confidence intervals are usually given.”

  • The paper relies on this without support.

    Incidence and prevalence studies often draw or report findings regarding the differences between groups.

    “Incidence and prevalence studies often draw or report findings regarding the differences between groups.”
15 of 15

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Each included study underwent quality assessment using the Joanna Briggs Institute (JBI) critical appraisal tools.

    “Each study included in the systematic review underwent a quality assessment to evaluate the research methodology employed in each study to ensure the reliability and validity of its findings. The Joanna Briggs Institute (JBI) critical appraisal tools, widely acknowledged and reliable for quality assessment, were used to investigate each study [17].”

    From Malaria in pregnancy: Meta-analyses of prevalence and associated complications (2024) · cites this paper as [17]

    The Developing the Tool and Conclusion sections identify a JBI critical appraisal tool intended for assessing studies in prevalence systematic reviews. However, the Discussion says its inter-rater reliability still needs testing, so this paper does not establish the citing sentence’s description of the tool as reliable. Whether the citing authors used it for every included study is their own methodological report, not a finding attributed to this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The Joanna Briggs Institute Critical Appraisal Checklist and MMAT have been previously validated.

    “both have been previously validated [23, 24].”

    From What are the barriers, facilitators and interventions targeting help-seeking behaviours for common mental health problems in adolescents? A systematic review (2020) · cites this paper as [23]

    The Pilot testing and Discussion sections report that the Joanna Briggs Institute prevalence checklist had face validity in a pilot test. The Discussion also says that its construct validity and inter-rater reliability still needed testing, so the unqualified claim that it had been validated overstates this source. This paper does not establish validation of MMAT. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Study quality was assessed with the Joanna Briggs Institute prevalence critical appraisal tool, with two reviewers independently conducting assessment and discrepancies resolved with a third reviewer.

    “Two reviewers independently conducted the quality assessment using the JBI tool, with any discrepancies resolved through discussion with a third reviewer.”

    From Adherence to PCSK9 Inhibitors in Clinical Practice (2026) · cites this paper as [18]

  • Two reviewers independently assessed risk of bias and study quality of each included study with the adjusted Joanna Briggs Institute Critical Appraisal Checklist for Studies Reporting Prevalence Data.

    “Two reviewers (MD and CS) independently assessed the risk of bias/ study quality of each included study with the adjusted Joanna Briggs Institute Critical Appraisal Checklist for Studies Reporting Prevalence Data (Munn et al., 2014).”

    From Prevalence of attention-deficit/hyperactivity disorder in older adults: a systematic review and meta-analysis (2020) · cites this paper as Munn et al., 2014

  • Tropical climates with appropriate temperature, humidity, and rainfall encourage endemicity of malaria because they are conducive to reproduction of the parasite within Anopheles mosquitoes, which are native to these environments.

    “Tropical climates with appropriate temperature, humidity, and rainfall conditions encourage endemicity of the disease as they are conducive to the reproduction of the parasite within the anopheles’ mosquito, which is itself native to these environments [17].”

    From Malaria in pregnancy: Meta-analyses of prevalence and associated complications (2024) · cites this paper as [17]

    The Introduction and the section on sample-size adequacy discuss measuring and appraising disease prevalence, not malaria transmission. Searches for malaria, Anopheles, climate conditions, and parasite reproduction found no passage addressing the claim. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. The Joanna Briggs Institute. Joanna Briggs Institute Reviewers' Manual: 2011 edition. Adelaide: The Joanna Briggs Institute; 2011.
  2. Munn Z, Moola S, Lisy K, Riitano D. The Systematic Review of Prevalence and Incidence Data The Joanna Briggs Institute Reviewer's Manual 2014. Australia: The Joanna Briggs Institute; 2014.

Cited by

Sorted by citations

Related

Authors
Munn, Z., Moola, S., Riitano, D., & Lisy, K.
Affiliations
The University of Adelaide (AU)
ORCID
Venue
International Journal of Health Policy and Management
Publisher
Maad Rayan Publishing Company
Published
2014
Volume
3
Issue
3
Pages
123-128
ISSN
2322-5939 (ISSN-L)
DOI
10.15171/ijhpm.2014.71
PMC
PMC4154549
OpenAlex
W2167191881
Semantic Scholar
834402d2ae3882ab962ec84179a5857fc9bd890b
Cited by
1,479
References
23
Citations by year
2026: 145 · 2025: 166 · 2024: 177 · 2023: 193 · 2022: 220 · 2021: 172 · 2020: 151 · 2019: 93 · 2018: 73 · 2017: 37 · 2016: 39 · 2015: 11 · 2014: 2
FWCI
8.12
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • International Journal of Health Policy and Management (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Research Information System of Ardabil University of Medical Sciences (Ardabil University of Medical Sciences) (acceptedVersion)
  • SSRN Electronic Journal (submittedVersion)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
critical appraisal tools, prevalence, systematic reviews, methodological quality assessment, prevalence data, tool development, pilot study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review