Cited by 261Meta-analysis and systematic reviewsHealth Education and ValidationHealth Sciences Research and Education

A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed

Crowe, Michael, Sheppard, Lorraine·2011·Journal of Clinical Epidemiology

Abstract

Objectives To evaluate critical appraisal tools (CATs) that have been through a peer-reviewed development process with the aim of analyzing well-designed, documented, and researched CATs that could be used to develop a comprehensive CAT.Study design and setting A critical review of the development of CATs was undertaken.Results Of the 44 CATs reviewed, 25 (57%) were applicable to more than one research design, 11 (25%) to true experimental studies, and the remaining 8 (18%) to individual research designs. Comprehensive explanation of how a CAT was developed and guidelines to use the CAT were available in five (11%) instances. There was no validation process reported in 11 CATs (25%) and 33 CATs (77%) had not been reliability tested. The questions and statements that made up each CAT were coded into 8 categories and 22 items such that each item was distinct from every other.Conclusions CATs are being developed while ignoring basic research techniques, the evidence available for design, and comprehensive validation and reliability testing. The basic structure for a comprehensive CAT is suggested that requires further study to verify its overall usefulness. Meanwhile, users of CATs should be careful about which CAT they use and how they use it.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2010.02.008

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Crowe, M., & Sheppard, L.
Affiliations
  • James Cook University (AU)
  • University of South Australia (AU)
Corresponding author
Crowe, Michael
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2011-01
Volume
64
Issue
1
Pages
79-89
ISSN
0895-4356 (ISSN-L)
OpenAlex
W2029463586
Cited by
261
References
75
Citations by year
2026: 17 · 2025: 16 · 2024: 13 · 2023: 13 · 2022: 24 · 2021: 32 · 2020: 16 · 2019: 26 · 2018: 18 · 2017: 24 · 2016: 14 · 2015: 12 · 2014: 13 · 2013: 10 · 2012: 5
FWCI
2.06
Type
journal article
Language
en
Open access
closed
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • ResearchOnline at James Cook University (James Cook University) (acceptedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Education › Health Education and Validation
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
critical appraisal tools, tool development, validation, reliability testing, research design, tool structure, systematic reviews, evidence-based medicine
MeSH terms
Humans, Research Design, Review Literature as Topic, Reproducibility of Results, Peer Review, Research, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review