Challenges in meta-analyses with observational studies
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Objective: Meta-analyses of observational studies are frequently published in the literature, but they are generally considered suboptimal to those involving randomised controlled trials (RCTs) only. This is due to the increased risk of biases that observational studies may entail as well as because of the high heterogeneity that might be present. In this article, we highlight aspects of meta-analyses with observational studies that need more careful consideration in comparison to meta-analyses of RCTs. Methods: We present an overview of recommendations from the literature with respect to how the different steps of a meta-analysis involving observational studies should be comprehensively conducted. We focus more on issues arising at the step of the quantitative synthesis, in terms of handling heterogeneity and biases. We briefly describe some sophisticated synthesis methods, which may allow for more flexible modelling approaches than common meta-analysis models. We illustrate the issues encountered in the presence of observational studies using an example from mental health, which assesses the risk of myocardial infarction in antipsychotic drug users. Results: The increased heterogeneity observed among studies challenges the interpretation of the diamond, while the inclusion of short exposure studies may lead to an exaggerated risk for myocardial infarction in this population. Conclusions: In the presence of observational study designs, prior to synthesis, investigators should carefully consider whether all studies at hand are able to answer the same clinical question. The potential for a quantitative synthesis should be guided through examination of the amount of clinical and methodological heterogeneity and assessment of possible biases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Cuschieri, S. (2019) · Saudi Journal of Anaesthesia · doi:10.4103/sja.sja_543_18 · Cited by 4,331 · cited 1×
“Although guidance on the reporting of observational studies does exist (eg, the Strengthening the Reporting of Observational Studies in Epidemiology checklist), 8 following this guidance is not mandatory yet for most medical journals.”
Analysing data and undertaking meta‐analyses
Deeks, J. J., Higgins, J. P., Altman, D. G., & on behalf of the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch10 · Cited by 2,672 · cited 1×
“21 Ideally, sources of heterogeneity and biases are better explored when individual participant data are available, and therefore, meta-analysts should seek for such data when possible.”
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · cited 1×
“4 Therefore, meta-analysis of observational data alone or in combination with RCTs (when possible) is often desirable.”
Analysing Data and Undertaking Meta‐Analyses
Deeks, J. J., Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch9 · Cited by 1,800
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
Meta-analysis of individual participant data: rationale, conduct, and reporting
Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010) · BMJ · doi:10.1136/bmj.c221 · Cited by 1,701 · cited 1×
“22 Nonetheless, most studies will in practice only provide aggregate data, and thus, clinical understanding of both the studies at hand and the assumptions of the different statistical approaches are important aspects for choosing plausible modelling strategies.”
Meta-analysis Spurious precision? Meta-analysis of observational studies
Egger, M., Schneider, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7125.140 · Cited by 997
Cited by
MOOSE Reporting Guidelines for Meta-analyses of Observational Studies
Brooke, B. S., Schwartz, T. A., & Pawlik, T. M. (2021) · JAMA Surgery · doi:10.1001/jamasurg.2021.0522 · Cited by 742
Roseborough, A. D., Saad, L., Goodman, M., Cipriano, L. E., Hachinski, V. C., & Whitehead, S. N. (2023) · Alzheimer's & Dementia · doi:10.1002/alz.12642 · Cited by 80
Metadata
- Authors
- Metelli, S., & Chaimani, A.
- Venue
- Evidence Based Mental Health
- Publisher
- BMJ
- Published
- 2020-03-05
- Volume
- 23
- Issue
- 2
- Pages
- 83-87
- ISSN
- 1362-0347 (ISSN-L)
- 1468-960X
- OpenAlex
- W3009392267
- Cited by
- 290
- References
- 23
- FWCI
- 24.44
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Challenges in meta-analyses with observational studies. https://app.cochat.ai/papers/10.1136/ebmental-2019-300129
- The paper
- Metelli, S., & Chaimani, A. (2020). Challenges in meta-analyses with observational studies. Evidence Based Mental Health. https://doi.org/10.1136/ebmental-2019-300129
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.