Extending an evidence hierarchy to include topics other than treatment: revising the Australian 'levels of evidence'
Open accessAbstract
BACKGROUND: In 1999 a four-level hierarchy of evidence was promoted by the National Health and Medical Research Council in Australia. The primary purpose of this hierarchy was to assist with clinical practice guideline development, although it was co-opted for use in systematic literature reviews and health technology assessments. In this hierarchy interventional study designs were ranked according to the likelihood that bias had been eliminated and thus it was not ideal to assess studies that addressed other types of clinical questions. This paper reports on the revision and extension of this evidence hierarchy to enable broader use within existing evidence assessment systems. METHODS: A working party identified and assessed empirical evidence, and used a commissioned review of existing evidence assessment schema, to support decision-making regarding revision of the hierarchy. The aim was to retain the existing evidence levels I-IV but increase their relevance for assessing the quality of individual diagnostic accuracy, prognostic, aetiologic and screening studies. Comprehensive public consultation was undertaken and the revised hierarchy was piloted by individual health technology assessment agencies and clinical practice guideline developers. After two and a half years, the hierarchy was again revised and commenced a further 18 month pilot period. RESULTS: A suitable framework was identified upon which to model the revision. Consistency was maintained in the hierarchy of "levels of evidence" across all types of clinical questions; empirical evidence was used to support the relationship between study design and ranking in the hierarchy wherever possible; and systematic reviews of lower level studies were themselves ascribed a ranking. The impact of ethics on the hierarchy of study designs was acknowledged in the framework, along with a consideration of how harms should be assessed. CONCLUSION: The revised evidence hierarchy is now widely used and provides a common standard against which to initially judge the likelihood of bias in individual studies evaluating interventional, diagnostic accuracy, prognostic, aetiologic or screening topics. Detailed quality appraisal of these individual studies, as well as grading of the body of evidence to answer each clinical, research or policy question, can then be undertaken as required.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Studies were ranked in accordance with Australia's National Health and Medical Research Council Evidence Hierarchy for Interventions.
“Studies were further ranked in accordance with Australia’s National Health and Medical Research Council (NHMRC) Evidence Hierarchy for Interventions [69, 70].”
From Incorporating animal-assisted therapy in mental health treatments for adolescents: A systematic review of canine assisted psychotherapy (2019) · cites this paper as [69]
Only four of the studies achieved 'fair' or 'good' methodological quality [65, 66] plus a moderate to high level of evidence [1, 69].
“Only four of the studies achieved ‘fair’ or ‘good’ methodological quality [65, 66] plus a moderate to high level of evidence [1, 69].”
From Incorporating animal-assisted therapy in mental health treatments for adolescents: A systematic review of canine assisted psychotherapy (2019) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252 · see citing works
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592 · see citing works
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563 · see citing works
What is ‘quality of evidence’ and why is it important to clinicians?
Guyatt, G. H., Oxman, A. D., Kunz, R., Vist, G. E., Falck-Ytter, Y., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39490.551019.be · Cited by 3,250 · see citing works
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347 · see citing works
Empirical evidence of design-related bias in studies of diagnostic tests.
Lijmer, J. G. (1999) · JAMA · doi:10.1001/jama.282.11.1061 · Cited by 1,861 · see citing works
Grading quality of evidence and strength of recommendations for diagnostic tests and strategies
Schünemann, H. J., Oxman, A. D., Brozek, J., Glasziou, P., Jaeschke, R., Vist, G. E., Williams, J. W., Kunz, R., Craig, J., Montori, V. M., et al. (2008) · BMJ · doi:10.1136/bmj.39500.677199.ae · Cited by 1,782 · see citing works
Why we need observational studies to evaluate the effectiveness of health care
Black, N. (1996) · BMJ · doi:10.1136/bmj.312.7040.1215 · Cited by 1,598 · see citing works
Systematic reviews of evaluations of diagnostic and screening tests
Deeks, J. J. (2001) · BMJ · doi:10.1136/bmj.323.7305.157 · Cited by 1,345 · see citing works
Atkins, D., Eccles, M., Flottorp, S., Guyatt, G. H., Henry, D., Hill, S., Liberati, A., O'Connell, D., Oxman, A. D., Phillips, B., et al. (2004) · BMC Health Services Research · doi:10.1186/1472-6963-4-38 · Cited by 1,235 · see citing works
Systematic reviews of evaluations of diagnostic and screening tests
(2001) · Europe PMC (PubMed Central) · Cited by 850 · see citing works
Kunz, R., & Oxman, A. D. (1998) · BMJ · doi:10.1136/bmj.317.7167.1185 · Cited by 770 · see citing works
Evidence base of clinical diagnosis: The architecture of diagnostic research
Sackett, D. L. (2002) · BMJ · doi:10.1136/bmj.324.7336.539 · Cited by 724 · see citing works
A hierarchy of evidence for assessing qualitative health research
Daly, J., Willis, K., Small, R., Green, J., Welch, N., Kealy, M., & Hughes, E. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.014 · Cited by 545 · see citing works
A Guide to the Development, Implementation and Evaluation of Clinical Practice Guidelines
(1999) · PsycEXTRA Dataset · doi:10.1037/e508692012-001 · Cited by 531 · see citing works
Heneghan, C., & Badenoch, D. (2006) · doi:10.1002/9780470750605.ch15 · Cited by 474 · see citing works
A comparison of observational studies and randomized, controlled trials
Benson, K., & Hartz, A. J. (2000) · American Journal of Ophthalmology · doi:10.1016/s0002-9394(00)00754-6 · Cited by 459 · see citing works
How to use the evidence: assessment and application of scientific evidence
Glasziou, P., Irwig, L., Bain, C., & Colditz, G. A. (2000) · Cited by 378 · see citing works
Atkins, D. C., Briss, P. A., Eccles, M., Flottorp, S. A., Guyatt, G. H., Harbour, R., Hill, S. R., Jaeschke, R., Liberati, A., Magrini, N., et al. (2005) · BMC Health Services Research · doi:10.1186/1472-6963-5-25 · Cited by 331 · see citing works
How to review the evidence: systematic identification and review of the scientific literature
Glasziou, P., Irwig, L., Bain, C., & Colditz, G. A. (2000) · Cited by 267 · see citing works
Critical appraisal of epidemiological studies and clinical trials
Elwood, M. (1998) · Cited by 220 · see citing works
PSYCHIATRIC INDICATIONS FOR THE TERMINATION OF PREGNANCY
Paulley, J. W. (1971) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1971.tb87854.x · Cited by 164 · see citing works
Egger, M., Ebrahim, S., & Smith, G. D. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.1 · Cited by 160 · see citing works
Chapter 20: Qualitative research and Cochrane reviews
Noyes, J., Popay, J., Pearson, A., Hannes, K., & Booth, A. (2008) · Cited by 96 · see citing works
Using socioeconomic evidence in clinical practice guidelines
Aldrich, R., Kemp, L., Williams, J. S., Harris, E., Simpson, S., Wilson, A., McGill, K., Byles, J., Lowe, J., & Jackson, T. J. (2003) · BMJ · doi:10.1136/bmj.327.7426.1283 · Cited by 92 · see citing works
Distressed or relieved? Psychological side effects of breast cancer screening in The Netherlands.
Scaf-Klomp, W., Sanderman, R., van de Wiel, H. B. M., et al. (1997) · Journal of Epidemiology & Community Health · doi:10.1136/jech.51.6.705 · Cited by 85 · see citing works
Systematic reviews of diagnostic tests in cancer: review of methods and reporting
Mallett, S. (2006) · BMJ · doi:10.1136/bmj.38895.467130.55 · Cited by 80 · see citing works
Bellomo, R., & Bagshaw, S. M. (2006) · Critical Care · doi:10.1186/cc5045 · Cited by 68 · see citing works
The ethics of screening: is 'screeningitis' an incurable disease?
Shickle, D., & CHADWICK, R. F. (1994) · Journal of Medical Ethics · doi:10.1136/jme.20.1.12 · Cited by 61 · see citing works
How to put the evidence into practice: implementation and dissemination strategies
Ragg, M., RUBIN, G. L., Frommer, M. S., et al. (2000) · Cited by 55 · see citing works
How to compare the costs and benefits: evaluation of the economic evidence
Gerard, K., Lowin, A., Schuffham, P., Haywood, P., & Hall, J. P. (2001) · ePrints Soton (University of Southampton) · Cited by 46 · see citing works
Newman, H., & Al‐Booz, H. (2008) · Clinical Oncology · doi:10.1016/j.clon.2008.08.003 · Cited by 42 · see citing works
Jackson, B. R. (2008) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2007.12.009 · Cited by 41 · see citing works
The Ethics of Placebo‐controlled Trials: A Comparison of Inert and Active Placebo Controls
Edward, S. J. L., Stevens, A. J., Braunholtz, D., Lilford, R. J., & Swift, T. L. (2005) · World Journal of Surgery · doi:10.1007/s00268-005-7621-7 · Cited by 38 · see citing works
Systematic reviews of qualitative evidence: what are the experiences of women with endometriosis?
Denny, E., & Khan, K. S. (2006) · Journal of Obstetrics and Gynaecology · doi:10.1080/01443610600797301 · Cited by 35 · see citing works
Safety Outcomes in Meta-analyses of Phase 2 vs Phase 3 Randomized Trials
Eikelboom, J. W., Mehta, S. R., Pogue, J., et al. (2001) · JAMA · doi:10.1001/jama.285.4.444 · Cited by 32 · see citing works
Moving beyond effectiveness : methodological issues in the synthesis of diverse sources of evidence.
Popay, J. (2006) · Lancaster EPrints (Lancaster University) · Cited by 24 · see citing works
Critical appraisal of epidemiological studies and clinical trials
Martínez‐González, M. Á. (2008) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2007.068643 · Cited by 22 · see citing works
Leung, G. M., Woo, P., Cowling, B. J., et al. (2008) · Journal of Public Health · doi:10.1093/pubmed/fdn034 · Cited by 14 · see citing works
(1983) · PubMed · Cited by 3 · see citing works
An empirical comparison of methods for meta-analysis of studies of diagnostic accuracy
Harbord, R., Bachmann, L. M., Shang, A., et al. (2005) · Bristol Research (University of Bristol) · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Merlin T Weston A Tooher R Revising a national standard: redevelopment of the Australian NHMRC evidence hierarchy Italian Journal of Public Health (Supplement 1) 2005 2 2 156
- Phillips B Ball C Sackett D Badenoch D Straus S Haynes B Dawes M Oxford Centre for Evidence-Based Medicine Levels of Evidence (May 2001) 2001 Oxford: Centre for Evidence-Based Medicine
Cited by
Sorted by citations
Lindeman, N. I., Cagle, P. T., Beasley, M. B., Chitale, D. A., Dacic, S., Giaccone, G., Jenkins, R. B., Kwiatkowski, D. J., Saldivar, J.-S., Squire, J., et al. (2013) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e318290868f · Cited by 1,280 · see citing works
Risk of recurrence after venous thromboembolism in men and women: patient level meta-analysis
Douketis, J., Tosetto, A., Marcucci, M., Baglin, T., Cosmi, B., Cushman, M., Kyrle, P., Poli, D., Tait, R. C., & Iorio, A. (2011) · BMJ · doi:10.1136/bmj.d813 · Cited by 278 · see citing works
Varker, T., Brand, R. M., Ward, J., Terhaag, S., & Phelps, A. (2019) · Psychological Services · doi:10.1037/ser0000239 · Cited by 261 · see citing works
Douketis, J., Tosetto, A., Marcucci, M., Baglin, T., Cushman, M., Eichinger, S., Palareti, G., Poli, D., Tait, R. C., & Iorio, A. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-8-201010190-00009 · Cited by 175 · see citing works
Jones, M. G., Rice, S. M., & Cotton, S. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210761 · Cited by 164 · see citing works
Meurk, C., Leung, J., Hall, W., Head, B. W., & Whiteford, H. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4827 · Cited by 105 · see citing works
Benzodiazepine use and aggressive behaviour: a systematic review
Albrecht, B., Staiger, P. K., Hall, K., Miller, P., Best, D., & Lubman, D. I. (2014) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867414548902 · Cited by 87 · see citing works
Do social determinants influence post-stroke aphasia outcomes? A scoping review
O'Halloran, R., Renton, J., Harvey, S., McSween, M.-P., & Wallace, S. J. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2193760 · Cited by 25 · see citing works
- Authors
- Merlin, T., Weston, A., & Tooher, R.
- Affiliations
- The University of Adelaide (AU)
- Corresponding author
- Merlin, Tracy
- ORCID
- Merlin, Tracy
- Venue
- BMC Medical Research Methodology
- Published
- 2009-06-11
- Volume
- 9
- Issue
- 1
- Pages
- 34-34
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC2700132
- OpenAlex
- W2044921283
- Cited by
- 435
- References
- 41
- Citations by year
- 2026: 9 · 2025: 23 · 2024: 16 · 2023: 39 · 2022: 41 · 2021: 41 · 2020: 33 · 2019: 35 · 2018: 37 · 2017: 27 · 2016: 24 · 2015: 17 · 2014: 22 · 2013: 28 · 2012: 10
- FWCI
- 17.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Health and Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- level of evidence, evidence hierarchy, bias assessment, diagnostic accuracy studies, prognostic studies, screening studies, health technology assessment, systematic reviews, evidence appraisal, research ethics
- MeSH terms
- Clinical Trials as Topic, Decision Support Techniques, Humans, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
