Visual representations of meta-analyses of multiple outcomes: Extensions to forest plots, funnel plots, and caterpillar plots
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Meta-analytic datasets can be large, especially when in primary studies multiple effect sizes are reported. The visualization of meta-analytic data is therefore useful to summarize data and understand information reported in primary studies. The gold standard figures in meta-analysis are forest and funnel plots. However, none of these plots can yet account for the existence of multiple effect sizes within primary studies. This manuscript describes extensions to the funnel plot, forest plot and caterpillar plot to adapt them to three-level meta-analyses. For forest plots, we propose to plot the study-specific effects and their precision, and to add additional confidence intervals that reflect the sampling variance of individual effect sizes. For caterpillar plots and funnel plots, we recommend to plot individual effect sizes and averaged study-effect sizes in two separate graphs. For the funnel plot, plotting separate graphs might improve the detection of both publication bias and/or selective outcome reporting bias.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342
Meta-Analysis in Clinical Research
L'ABBÉ, K. A., DETSKY, A. S., & O'ROURKE, K. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-107-2-224 · Cited by 1,267
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130 · cited 2×
“…plot might be symmetric, but it could be the case that studies with large positive effects reported fewer outcomes in comparison with studies that reported on average smaller or even negative effects, which would again be an indication of reporting bias (Van den Noortgate et al., 2013).”
MethodResultAdjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Fixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450
Cited by
Traut, H. J., Guild, R. M., & Munakata, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.662139 · Cited by 100
Llanos-Lagos, C., Ramirez-Campillo, R., Moran, J., & Sáez de Villarreal, E. (2024) · Sports Medicine · doi:10.1007/s40279-023-01978-y · Cited by 68
Warneke, K., Plöschberger, G., Lohmann, L. H., Lichtenstein, E., Jochum, D., Siegel, S. D., Zech, A., & Behm, D. G. (2024) · Journal of Sport and Health Science · doi:10.1016/j.jshs.2024.01.006 · Cited by 42
“The risk of publication bias was quantified by the Egger’s regression test in addition to a visual inspection of modified funnel plots with the extension for dependent effect sizes (ESs) 35 to account for multiple study outcomes.”
MethodDing, Z., Leung, P.-Y., Lee, T.-L., & Chan, A. S. (2023) · Ageing Research Reviews · doi:10.1016/j.arr.2023.101886 · Cited by 31
Dreisoerner, A., Goetz, V., Frohnmayer, D., Tran, U. S., Voracek, M., & Nater, U. M. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02505-5 · Cited by 2
“Publication bias was visually assessed via funnel plots for multilevel meta-analysis using the code provided by Fernández-Castilla and colleagues 243 , and formally with the three-level version of Egger’s regression test 244,245 .”
Metadata
- Authors
- Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W.
- Venue
- Methodology
- Published
- 2020-12-22
- Volume
- 16
- Issue
- 4
- Pages
- 299-315
- ISSN
- 1614-1881 (ISSN-L)
- 1614-2241
- OpenAlex
- W3115094058
- Cited by
- 127
- References
- 13
- FWCI
- 5.45
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.