CoChat
Sign Up
Cited by 534Meta-analysis and systematic reviewsAdvanced Causal Inference TechniquesStatistical Methods in Epidemiology

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials

Open access
Anglemyer, Andrew, Horvath, Hacsi T, Bero, Lisa·2014·Cochrane Database of Systematic Reviews
DOI 10.1002/14651858.mr000034.pub2PMC PMC8191367
Open this paper in CoChat — free account

Abstract

BACKGROUND: Researchers and organizations often use evidence from randomized controlled trials (RCTs) to determine the efficacy of a treatment or intervention under ideal conditions. Studies of observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and modifications of existing designs, including both randomized and observational, are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population.A systematic analysis of study design features, risk of bias, parameter interpretation, and effect size for all types of randomized and non-experimental observational studies is needed to identify specific differences in design types and potential biases. This review summarizes the results of methodological reviews that compare the outcomes of observational studies with randomized trials addressing the same question, as well as methodological reviews that compare the outcomes of different types of observational studies. OBJECTIVES: To assess the impact of study design (including RCTs versus observational study designs) on the effect measures estimated.To explore methodological variables that might explain any differences identified.To identify gaps in the existing research comparing study designs. SEARCH METHODS: We searched seven electronic databases, from January 1990 to December 2013.Along with MeSH terms and relevant keywords, we used the sensitivity-specificity balanced version of a validated strategy to identify reviews in PubMed, augmented with one term ("review" in article titles) so that it better targeted narrative reviews. No language restrictions were applied. SELECTION CRITERIA: We examined systematic reviews that were designed as methodological reviews to compare quantitative effect size estimates measuring efficacy or effectiveness of interventions tested in trials with those tested in observational studies. Comparisons included RCTs versus observational studies (including retrospective cohorts, prospective cohorts, case-control designs, and cross-sectional designs). Reviews were not eligible if they compared randomized trials with other studies that had used some form of concurrent allocation. DATA COLLECTION AND ANALYSIS: In general, outcome measures included relative risks or rate ratios (RR), odds ratios (OR), hazard ratios (HR). Using results from observational studies as the reference group, we examined the published estimates to see whether there was a relative larger or smaller effect in the ratio of odds ratios (ROR).Within each identified review, if an estimate comparing results from observational studies with RCTs was not provided, we pooled the estimates for observational studies and RCTs. Then, we estimated the ratio of ratios (risk ratio or odds ratio) for each identified review using observational studies as the reference category. Across all reviews, we synthesized these ratios to get a pooled ROR comparing results from RCTs with results from observational studies. MAIN RESULTS: Our initial search yielded 4406 unique references. Fifteen reviews met our inclusion criteria; 14 of which were included in the quantitative analysis.The included reviews analyzed data from 1583 meta-analyses that covered 228 different medical conditions. The mean number of included studies per paper was 178 (range 19 to 530).Eleven (73%) reviews had low risk of bias for explicit criteria for study selection, nine (60%) were low risk of bias for investigators' agreement for study selection, five (33%) included a complete sample of studies, seven (47%) assessed the risk of bias of their included studies,Seven (47%) reviews controlled for methodological differences between studies,Eight (53%) reviews controlled for heterogeneity among studies, nine (60%) analyzed similar outcome measures, and four (27%) were judged to be at low risk of reporting bias.Our primary quantitative analysis, including 14 reviews, showed that the pooled ROR comparing effects from RCTs with effects from observational studies was 1.08 (95% confidence interval (CI) 0.96 to 1.22). Of 14 reviews included in this analysis, 11 (79%) found no significant difference between observational studies and RCTs. One review suggested observational studies had larger effects of interest, and two reviews suggested observational studies had smaller effects of interest.Similar to the effect across all included reviews, effects from reviews comparing RCTs with cohort studies had a pooled ROR of 1.04 (95% CI 0.89 to 1.21), with substantial heterogeneity (I(2) = 68%). Three reviews compared effects of RCTs and case-control designs (pooled ROR: 1.11 (95% CI 0.91 to 1.35)).No significant difference in point estimates across heterogeneity, pharmacological intervention, or propensity score adjustment subgroups were noted. No reviews had compared RCTs with observational s

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/14651858.mr000034.pub2

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 2

  1. Deeks JJ D'Amico R Sakarovitch C Are comparability of case-mix and the use of statistical adjustment markers of quality in non-randomised studies? An empirical investigation 4th Symposium on Systematic Reviews: Pushing the Boundaries Oxford, UK 2002
  2. Shepherd J Bagnall A Colquitt J 'Sometimes similar, sometimes different': a systematic review of meta-analyses of randomised and non-randomised policy intervention studies 14th Cochrane Colloquium, Dublin, Ireland

Cited by

Sorted by citations

Items 401–500 of 532

Related

See all

Metadata

Authors
Anglemyer, A., Horvath, H. T., & Bero, L.
Affiliations
  • University of California, San Francisco (US)
  • University of California, San Francisco; Global Health Sciences; San Francisco California USA 94105
ORCID
Venue
Cochrane Database of Systematic Reviews
Publisher
Wiley
Published
2014-04-29
Volume
2014
Issue
4
Pages
MR000034-MR000034
ISSN
  • 1361-6137 (ISSN-L)
  • 1465-1858
DOI
10.1002/14651858.mr000034.pub2
PMC
PMC8191367
OpenAlex
W1972783815
Cited by
534
References
74
Citations by year
2026: 10 · 2025: 21 · 2024: 37 · 2023: 45 · 2022: 63 · 2021: 73 · 2020: 60 · 2019: 50 · 2018: 43 · 2017: 49 · 2016: 41 · 2015: 36 · 2014: 6
FWCI
43.06
Type
journal article
Language
en
Update notices
new_version (2024-01-04)
Open access
Locations
  • Cochrane Database of Systematic Reviews (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
randomized controlled trials, observational studies, comparative effectiveness research, effect size estimation, risk of bias, study design comparison, relative risk, odds ratio, hazard ratio, retrospective cohort, prospective cohort, case-control study, cross-sectional study, methodological review, real-world effectiveness, bias in observational studies, effect size heterogeneity
MeSH terms
Humans, Meta-Analysis as Topic, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Observational Studies as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review