CoChat
Sign Up
Cited by 107Meta-analysis and systematic reviewsAdvanced Causal Inference TechniquesStatistical Methods in Epidemiology

The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution

Open access
Dreyer, Nancy A., Velentgas, Priscilla, Westrich, Kimberly, Dubois, Robert·2014·Journal of Managed Care Pharmacy
DOI 10.18553/jmcp.2014.20.3.301PMC PMC10437555
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: While there is growing demand for information about comparative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making. OBJECTIVE: To develop and test an item checklist that can be used to qualify those observational CE studies sufficiently rigorous in design and execution to contribute meaningfully to the evidence base for decision support. METHODS: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external quality ratings (N=88 articles). The articles compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review-quality assessment from a published systematic review; (b) Single Expert Review-quality assessment made according to the solicited "expert opinion" of a senior researcher; and (c) Concordant Expert Review-quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare testers' assessments with those of experts. RESULTS: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better. CONCLUSIONS: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias. The current testing and validation efforts did not achieve clear discrimination between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a specific granular decision for evaluation, or not identifying a single study objective in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question. Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on studies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evaluate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18553/jmcp.2014.20.3.301

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U. Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report-Part III. Value Health. 2009;12(8):1062-73.
  2. Berger ML, Mamdani M, Atkins D, Johnson ML. Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report-Part I. Value Health. 2009;12(8):1044-52.
  3. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions- Agency for Healthcare Research and Quality and the Effective Health-Care Program. J Clin Epidemiol. 2010;63(5):513-23.
  4. McDonagh MS, Peterson K, Carson S, Fu R, Thakurta S. Drug class review: atypical antipsychotic drugs. Final Update 3 Report [Internet].
  5. Humphrey LL, Chan BK, Sox HC. Postmenopausal hormone replacement therapy and the primary prevention of cardiovascular disease. Ann Intern Med. 2002;137(4):273-84.
  6. Nelson HD, Humphrey LL, Nygren P, Teutsch SM, Allan JD. Postmenopausal hormone replacement therapy: scientific review. JAMA. 2002;288(7):872-81.
  7. Yank V, Tuohy CV, Logan AC, et al. Comparative effectiveness of in- hospital use of recombinant factor VIIa for off-label indications vs. usual care. Comparative Effectiveness Review No. 21. (Prepared by Stanford- UCSF Evidence-based Practice Center under Contract No. #290-02-0017).
  8. Lang S, Kleijnen J. Quality assessment tools for observational studies: lack of consensus. Int J Evid Based Healthc. 2010;8(4):247.
  9. Poor Bond GR, Kim HW, Meyer PS, et al. Response to vocational rehabilitation during treatment with first-or second-generation antipsychotics. Psychiatr Serv. 2004;55(1):59-66.
  10. Poor de Haan L, Beuk N, Hoogenboom B, Dingemans P, Linszen D. Obsessive-compulsive symptoms during treatment with olanzapine and risperidone: a prospective study of 113 patients with recent-onset schizophrenia or related disorders. J Clin Psychiatry. 2002;63(2):104-107.
  11. Poor Meyer JM. A retrospective comparison of weight, lipid, and glucose changes between risperidone-and olanzapine-treated inpatients: metabolic outcomes after 1 year. J Clin Psychiatry. 2002;63(5):425-33.
  12. Pelagotti F, Santarlasci B, Vacca F, Trippoli S, Messori A. Dropout rates with olanzapine or risperidone: a multi-centre observational study. Eur J Clin Pharmacol. 2004;59(12):905-09.
  13. Soholm B, Lublin H. Long-term effectiveness of risperidone and olanzapine in resistant or intolerant schizophrenic patients. A mirror study. Acta Psychiatr Scand. 2003;107(5):344-50.
  14. Poor Fernando HC, Schauer PR, Rosenblatt M, et al. Quality of life after antireflux surgery compared with nonoperative management for severe gastroesophageal reflux disease. J Am Coll Surg. 2002;194(1):23-27.
  15. Gelsomino S, Lorusso R, Romagnoli S, et al. Treatment of refractory bleeding after cardiac operations with low-dose recombinant activated factor VII (NovoSeven): a propensity score analysis. Eur J Cardiothorac Surg. 2008;33(1):64-71.
  16. Hallevi H, Gonzales NR, Barreto AD, et al. The effect of activated factor VII for intracerebral hemorrhage beyond 3 hours versus within 3 hours. Stroke. 2008;39(2):473-75.
  17. Poor Harrison TD, Laskosky J, Jazaeri O, et al. "Low-dose" recombinant activated factor VII results in less blood and blood product use in traumatic hemorrhage. J Trauma. 2005;59(1):150-54.
  18. Poor Holzman MD, Mitchel EF, Ray WA, Smalley WE. Use of health care resources among medically and surgically treated patients with gastroesophageal reflux disease: a population-based study. J Am Coll Surg. 2001;192(1):17-24.
  19. Poor Hyrich KL, Symmons DP, Watson KD, et al. Comparison of the response to infliximab or etanercept monotherapy with the response to cotherapy with methotrexate or another disease-modifying antirheumatic drug in patients with rheumatoid arthritis: results from the British Society for Rheumatology Biologics Register. Arthritis Rheum. 2006;54(6):1786-94.
  20. Kalicinski P, Markiewicz M, Kaminski A, et al. Single pretransplant bolus of recombinant activated factor VII ameliorates influence of risk factors for blood loss during orthotopic liver transplantation. Pediatr Transplant. 2005;9(3):299-304.
  21. Poor Setoguchi S, Solomon DH, Weinblatt ME, et al. Tumor necrosis factor alpha antagonist use and cancer in patients with rheumatoid arthritis. Arthritis Rheum. 2006;54(9):2757-64.
  22. Good Tran T, Spechler SJ, Richardson P, et al. Fundoplication and the risk of esophageal cancer in gastroesophageal reflux disease: a Veterans Affairs cohort study. Am J Gastroenterol. 2005;100(5):1002-08.
  23. Good Bush TL, Barrett-Connor E, Cowan LD, et al. Cardiovascular mortality and noncontraceptive use of estrogen in women: results from the Lipid Research Clinics Program Follow-up Study. Circulation. 1987;75(6):1102-09.
  24. Good Ettinger B, Friedman GD, Bush T, et al. Reduced mortality associated with long-term postmenopausal estrogen therapy. Obstet Gynecol. 1996;87(1):6-12.
  25. Good Grodstein F, Manson JE, Colditz GA, et al. A prospective, observational study of postmenopausal hormone therapy and primary prevention of cardiovascular disease. Ann Intern Med. 2000;133(12):933-41.
  26. Good Henderson BE, Paganini-Hill A, Ross RK. Decreased mortality in users of estrogen replacement therapy. Arch Intern Med. 1991;151(1):75-78.
  27. Poor Lafferty FW, Fiske ME. Postmenopausal estrogen replacement: a long-term cohort study. Am J Med. 1994;97(1):66-77.
  28. Poor Persson I, Yuen J, Bergkvist L, et al. Cancer incidence and mortality in women receiving estrogen and estrogen-progestin replacement therapy-long-term follow-up of a Swedish cohort. Int J Cancer. 1996;67(3):327-32.
  29. Good Colditz GA, Hankinson SE, Hunter DJ, et al. The use of estrogens and progestins and the risk of breast cancer in postmenopausal women. N Engl J Med. 1995;332(24):1589-93.
  30. Good Paganini-Hill A, Henderson VW. Estrogen replacement therapy and risk of Alzheimer disease. Arch Intern Med. 1996;156(19):2213-17.
  31. Poor Schairer C, Gail M, Byrne C, et al. Estrogen replacement therapy and breast cancer survival in a large screening study. J Natl Cancer Inst. 1999;91(3):264-70.
  32. Sellers TA, Mink PJ, Cerhan JR, et al. The role of hormone replacement therapy in the risk for breast cancer and total mortality in women with a family history of breast cancer. Ann Intern Med. 1997;127(11):973-80. Good Acute myocardial infarction and combined oral contraceptives: results of an international multicentre case-control study. WHO Collaborative Study of Cardiovascular Disease and Steroid Hormone Contraception. Lancet. 1997;349(9060):1202-09.
  33. Good APPENDix A Articles Tested from Systematic Reviews (continued) Article Reference Quality Single Expert Review Concordant Review Solomon DH, Massarotti E, Garg R, et al. Association between disease-modifying antirheumatic drugs and diabetes risk in patients with rheumatoid arthritis and psoriasis. JAMA. 2011;305(24):2525-31.
  34. Good Good Kim SY, Schneeweiss S, Katz JN, et al. Oral bisphosphonates and risk of subtrochanteric or diaphyseal femur fractures in a population-based cohort. J Bone Miner Res. 2011;26(5):993-1001.
  35. reviewed Srivastava R, Downey EC, O'Gorman M, et al. Impact of fundoplication versus gastrojejunal feeding tubes on mortality and in preventing aspiration pneumonia in young children with neurologic impairment who have gastroesophageal reflux disease. Pediatrics. 2009;123(1):338-45.
  36. Good Not reviewed Shishehbor MH, Hawi R, Singh IM, et al. Drug-eluting versus bare-metal stents for treating saphenous vein grafts. Am Heart J. 2009;158(4):637-43.
  37. Good Not reviewed Applegate RJ, Sacrinty MT, Kutcher MA, et al. Comparison of drug-eluting versus bare metal stents on later frequency of acute myocardial infarction and death. Am J Cardiol. 2007;99(3):333-38.
  38. Poor Poor Mayor M, Malik AZ, Minor RJ Jr, et al. One-year outcomes from the TAXUS express stent versus cypher stent. Am J Cardiol. 2009;103(7):930-36.
  39. Good Poor Kamat SA, Gandhi SK, Davidson M. Comparative effectiveness of rosuvastatin versus other statin therapies in patients at increased risk of failure to achieve low-density lipoprotein goals. Curr Med Res Opin. 2007;23(5):1121-30.
  40. Poor Not reviewed Wu E, Greenberg PE, Yang E, et al. Comparison of escitalopram versus citalopram for the treatment of major depressive disorder in a geriatric population. Curr Med Res Opin. 2008;24(9):2587-95.
  41. Poor Good Kubiak DW, Bryar JM, McDonnell AM, et al. Evaluation of caspofungin or micafungin as empiric antifungal therapy in adult patients with persistent febrile neutropenia: a retrospective, observational, sequential cohort analysis. Clin Ther. 2010;32(4):637-48.
  42. Poor Not reviewed Ramanath VS, Brown JR, Malenka DJ, et al; Dartmouth Dynamic Registry Investigators. Outcomes of diabetics receiving bare- metal stents versus drug-eluting stents. Catheter Cardiovasc Interv. 2010;76(4):473-81.
  43. Poor Not reviewed Shorr AF, Sarnes MW, Peeples PJ, et al. Comparison of cost, effectiveness, and safety of injectable anticoagulants used for thromboprophylaxis after orthopedic surgery. Am J Health Syst Pharm. 2007;64(22):2349-55.
  44. Good Poor Carragee EJ, Spinnickie AO, Alamin TF, et al. A prospective controlled study of limited versus subtotal posterior discectomy: short-term outcomes in patients with herniated lumbar intervertebral discs and large posterior anular defect. Spine (Phila Pa 1976). 2006;31(6):653-57.
  45. Poor Poor Chu WW, Kuchulakanti PK, Torguson R, et al. Comparison of clinical outcomes of overlapping sirolimus-versus paclitaxel- eluting stents in patients undergoing percutaneous coronary intervention. Am J Cardiol. 2006;98(12):1563-66.
  46. Poor Not reviewed Villareal RP, Lee VV, Elayda MA, et al. Coronary artery bypass surgery versus coronary stenting: risk-adjusted survival rates in 5,619 patients. Tex Heart Inst J. 2002;29(1):3-9.
  47. Good Poor Wolfe F, Michaud K, Stephenson B, et al. Toward a definition and method of assessment of treatment failure and treatment effectiveness: the case of leflunomide versus methotrexate. J Rheumatol. 2003;30(8):1725-32.

Cited by

Sorted by citations

Related

Authors
Dreyer, N. A., Velentgas, P., Westrich, K., & Dubois, R.
Affiliations
IQVIA (United Kingdom) (GB)
ORCID
Venue
Journal of Managed Care Pharmacy
Publisher
Academy of Managed Care Pharmacy
Published
2014-03
Volume
20
Issue
3
Pages
301-308
ISSN
  • 1083-4087 (ISSN-L)
  • 1944-706X
DOI
10.18553/jmcp.2014.20.3.301
PMC
PMC10437555
OpenAlex
W300479524
Cited by
107
References
12
Citations by year
2026: 1 · 2025: 2 · 2024: 5 · 2023: 6 · 2022: 5 · 2021: 12 · 2020: 13 · 2019: 6 · 2018: 14 · 2017: 19 · 2016: 14 · 2015: 8 · 2014: 2
FWCI
14.04
Type
journal article
Language
en
Open access
Locations
  • Journal of Managed Care Pharmacy (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Pfizer
  • AstraZeneca
  • University of Washington
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
observational studies, comparative effectiveness, study quality assessment, quality ratings, generalizability, bias, sensitivity analysis, positive and negative predictive value, evidence-based decision making, medical devices, drug safety, clinical intervention, surgical treatment, covariate selection, residual bias, health technology assessment, health policy decision support
MeSH terms
Decision Making, Humans, Research Design, Comparative Effectiveness Research, Checklist, Observational Studies as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review