An empirical comparison of Bayesian modelling strategies for missing binary outcome data in network meta-analysis
Open accessAbstract
BACKGROUND: A number of strategies have been proposed to handle missing binary outcome data (MOD) in systematic reviews. However, none of these have been evaluated empirically in a series of published systematic reviews. METHODS: Using published systematic reviews with network meta-analysis (NMA) from a wide range of health-related fields, we evaluated comparatively the most frequently described Bayesian modelling strategies for MOD in terms of log odds ratio (log OR), between-trial variance, inconsistency factor (i.e. difference between direct and indirect estimates for a comparison), surface under the cumulative ranking (SUCRA) and rankings. We extended the Bayesian random-effects NMA model to incorporate the informative missingness odds ratio (IMOR) parameter, and applied the node-splitting approach to investigate inconsistency locally. We considered both pattern-mixture and selection models, different structures for prior distribution of log IMOR, and different scenarios for MOD. To illustrate level of agreement between different strategies and scenarios, we used Bland-Altman plots. RESULTS: Addressing MOD using extreme scenarios and ignoring the uncertainty about the scenarios led to systematically different and more precise log ORs compared to modelling MOD under the missing at random (MAR) assumption. Hierarchical structure of log IMORs led to lower between-trial variance, especially in the case of substantial MOD. Assuming common-within-network or trial-specific log IMORs yielded similar posterior results for all NMA estimates, whereas intervention-specific structure systematically inflated uncertainty around log ORs and SUCRAs. Pattern-mixture model agreed with selection model, particularly under the trial-specific structure; however, selection model systematically reduced precision around log IMORs. Overall, different strategies and scenarios mostly had good agreement in the case of low MOD. CONCLUSIONS: Addressing MOD using extreme scenarios and/or ignoring the uncertainty about the scenarios may negatively affect NMA estimates. Modelling MOD via the IMOR parameter can ensure bias-adjusted estimates and offer valuable insights into missingness mechanisms. The researcher should seek an expert opinion in order to decide on the structure of log IMOR that best aligns to the condition and interventions studied and to define a proper prior distribution for log IMOR. Our findings also apply to pairwise meta-analyses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Measuring agreement in method comparison studies
Bland, J. M., & Altman, D. G. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800204 · Cited by 9,619 · see citing works
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950 · see citing works
Salanti, G., Ades, A. E., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.016 · Cited by 3,766 · see citing works
Checking consistency in mixed treatment comparison meta-analysis
Dias, S., Welton, N. J., Caldwell, D. M., & Ades, A. E. (2010) · Statistics in Medicine · doi:10.1002/sim.3767 · Cited by 2,521 · see citing works
Dias, S., Sutton, A. J., Ades, A. E., & Welton, N. J. (2013) · Medical Decision Making · doi:10.1177/0272989x12458724 · Cited by 1,207 · see citing works
van Valkenhoef, G., Dias, S., Ades, A. E., & Welton, N. J. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1167 · Cited by 699 · see citing works
Evidence Based Medicine—How to Practice and Teach EBM
Jones, K. (1996) · BMJ · doi:10.1136/bmj.313.7069.1410 · Cited by 521 · see citing works
Lambert, P. C., Sutton, A. J., Burton, P. R., Abrams, K. R., & Jones, D. R. (2005) · Statistics in Medicine · doi:10.1002/sim.2112 · Cited by 511 · see citing works
Higgins, J. P., Deeks, J. J., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch16 · Cited by 475 · see citing works
Turner, R. M., Jackson, D., Wei, Y., Thompson, S. G., & Higgins, J. P. T. (2015) · Statistics in Medicine · doi:10.1002/sim.6381 · Cited by 410 · see citing works
GetReal in network meta-analysis: a review of the methodology
Efthimiou, O., Debray, T. P. A., van Valkenhoef, G., Trelle, S., Panayidou, K., Moons, K. G. M., Reitsma, J. B., Shang, A., Salanti, G., & on behalf of GetReal Methods Review Group (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1195 · Cited by 401 · see citing works
Imputation methods for missing outcome data in meta-analysis of clinical trials
Higgins, J. P. T., White, I. R., & Wood, A. (2008) · Clinical Trials · doi:10.1177/1740774508091600 · Cited by 352 · see citing works
Including all individuals is not enough: lessons for intention-to-treat analysis
White, I., Carpenter, J., & Horton, N. (2012) · Clinical Trials · doi:10.1177/1740774512450098 · Cited by 299 · see citing works
Uncertainty method improved on best–worst case analysis in a binary meta-analysis
Gamble, C. L., & Hollis, S. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.09.013 · Cited by 239 · see citing works
Choice of effect measure for epidemiological data
Walter, S. D. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00210-9 · Cited by 193 · see citing works
Guyatt, G. H., Ebrahim, S., Alonso-Coello, P., Johnston, B. C., Mathioudakis, A. G., Briel, M., Mustafa, R. A., Sun, X., Walter, S. D., Heels-Ansdell, D., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.005 · Cited by 163 · see citing works
Akl, E. A., Johnston, B. C., Alonso‐Coello, P., Neumann, I., Ebrahim, S., Briel, M., Cook, D. J., & Guyatt, G. H. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057132 · Cited by 156 · see citing works
Ebrahim, S., Akl, E. A., Mustafa, R. A., Sun, X., Walter, S. D., Heels-Ansdell, D., Alonso-Coello, P., Johnston, B. C., & Guyatt, G. H. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.014 · Cited by 141 · see citing works
Πετροπούλου, Μ., Nikolakopoulou, A., Veroniki, A. A., Rios, P. D. L., Vafaei, A., Zarin, W., Gianatsi, M., Sullivan, S. M., Tricco, A. C., Chaimani, A., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.11.002 · Cited by 135 · see citing works
Characteristics of Networks of Interventions: A Description of a Database of 186 Published Networks
Nikolakopoulou, A., Chaimani, A., Veroniki, A. A., Vasiliadis, H. S., Schmid, C. H., & Salanti, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086754 · Cited by 127 · see citing works
Allowing for uncertainty due to missing data in meta‐analysis—Part 1: Two‐stage methods
White, I. R., Higgins, J. P. T., & Wood, A. (2007) · Statistics in Medicine · doi:10.1002/sim.3008 · Cited by 114 · see citing works
Streamlined Plot Theme and Plot Annotations for 'ggplot2'
Wilke, C. O. (2015) · Cited by 100 · see citing works
Lee, A. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.07.014 · Cited by 90 · see citing works
Mavridis, D., White, I. R., Higgins, J. P. T., Cipriani, A., & Salanti, G. (2014) · Statistics in Medicine · doi:10.1002/sim.6365 · Cited by 89 · see citing works
Akl, E. A., Kahale, L. A., Agoritsas, T., Brignardello-Petersen, R., Busse, J. W., Carrasco-Labra, A., Ebrahim, S., Johnston, B. C., Neumann, I., Sola, I., et al. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0083-6 · Cited by 66 · see citing works
Evaluating the impact of imputations for missing participant outcome data in a network meta-analysis
Spineli, L. M., Higgins, J. P. T., Cipriani, A., Leucht, S., & Salanti, G. (2013) · Clinical Trials · doi:10.1177/1740774512470317 · Cited by 49 · see citing works
Self-citationAllowing for uncertainty due to missing data in meta‐analysis—Part 2: Hierarchical models
White, I. R., Welton, N. J., Wood, A., Ades, A. E., & Higgins, J. P. T. (2007) · Statistics in Medicine · doi:10.1002/sim.3007 · Cited by 49 · see citing works
Turner, N. L., Dias, S., Ades, A. E., & Welton, N. J. (2015) · Statistics in Medicine · doi:10.1002/sim.6475 · Cited by 41 · see citing works
An Assessment of the Methodological Quality of Published Network Meta-Analyses: A Systematic Review
Chambers, J. D., Naci, H., Wouters, O. J., Pyo, J., Gunjal, S., Kennedy, I. R., Hoey, M. G., Winn, A. N., & Neumann, P. J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0121715 · Cited by 37 · see citing works
Meta‐Analysis of Studies with Missing Data
Yuan, Y., & Little, R. J. A. (2009) · Biometrics · doi:10.1111/j.1541-0420.2008.01068.x · Cited by 27 · see citing works
Magder, L. S. (2003) · Controlled Clinical Trials · doi:10.1016/s0197-2456(03)00021-7 · Cited by 27 · see citing works
Spineli, L. M., Yepes-Núñez, J. J., & Schünemann, H. J. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0576-9 · Cited by 22 · see citing works
Self-citationMeta-analysis with Missing Data
White, I. R., & Higgins, J. P. T. (2009) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900104 · Cited by 18 · see citing works
Spineli, L. M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.09.002 · Cited by 17 · see citing works
Self-citationSpineli, L. M. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1268 · Cited by 10 · see citing works
Self-citationDimitrakopoulou, V., Efthimiou, O., Leucht, S., & Salanti, G. (2014) · Statistics in Medicine · doi:10.1002/sim.6364 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Kahale, L. A., Khamis, A. M., Diab, B., Chang, Y., Lopes, L. C., Agarwal, A., Li, L., Mustafa, R. A., Koujanian, S., Waziry, R., et al. (2020) · BMJ · doi:10.1136/bmj.m2898 · Cited by 33 · see citing works
Li, M., Liu, S., Yu, B., Li, N., Lyu, A., Yang, H., He, H., Zhang, N., Ma, J., Sun, M., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/63209 · Cited by 12 · see citing works
“Differences between direct and indirect coefficients (assessed via the P value) were used to estimate inconsistency: if P .05, local inconsistency was considered to exist [46].”
- Authors
- Spineli, L. M.
- Affiliations
- Medizinische Hochschule Hannover (DE)
- Corresponding author
- Spineli, Loukia M.
- ORCID
- Spineli, Loukia M.
- Venue
- BMC Medical Research Methodology
- Published
- 2019-04-24
- Volume
- 19
- Issue
- 1
- Pages
- 86-86
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC6480793
- OpenAlex
- W2947696205
- Cited by
- 47
- References
- 38
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 5 · 2023: 3 · 2022: 5 · 2021: 11 · 2020: 7 · 2019: 1 · 2018: 1
- FWCI
- 4.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Deutsche Forschungsgemeinschaft (SP 1664/1-1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- network meta-analysis, pattern-mixture models, selection model, missing at random, node splitting, surface under the cumulative ranking curve
- MeSH terms
- Network Meta-Analysis as Topic, Systematic Reviews as Topic, Algorithms, Bayes Theorem, Humans, Models, Theoretical, Research Design, Reproducibility of Results, Odds Ratio, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
