Pitfalls of Systematic Reviews and Meta-Analyses in Imaging Research
Open accessThe DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Systematic reviews of imaging research represent a tool to better understand test accuracy or the efficacy of interventions. Like any type of research, appropriate methods must be applied to optimize quality. The purpose of this review is to outline common pitfalls in performing systematic reviews of imaging research, with a focus on challenges particular to performing reviews of diagnostic accuracy studies. The following challenges are highlighted: posing relevant review questions, conducting comprehensive literature searches, assessing for bias in included studies, testing for heterogeneity and publication bias, pooling results across studies, and forming appropriate conclusions. By guiding authors on how to overcome these, the hope is that published reviews in imaging research will be of higher quality and have a positive impact on clinical practice. In addition, the review aims to educate readers of reviews so they become aware of crucial elements of systematic reviews that could bias review results.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Self-citationReitsma, J. B., Glas, A. S., Rutjes, A. W. S., Scholten, R. J. P. M., Bossuyt, P. M., & Zwinderman, A. H. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.02.022 · Cited by 3,369
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Self-citationTowards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Self-citationDeeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Systematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
Morrison, A., Polisena, J., Husereau, D., Moulton, K., Clark, M., Fiander, M., Mierzwinski-Urban, M., Clifford, T., Hutton, B., & Rabb, D. (2012) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462312000086 · Cited by 1,257
Meta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,250
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Systematic Reviews of Diagnostic Test Accuracy
Leeflang, M. M. G., Deeks, J. J., Gatsonis, C., & Bossuyt, P. M. M. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-149-12-200812160-00008 · Cited by 1,124
Self-citationDirection and impact of language bias in meta-analyses of controlled trials: empirical study
Jüni, P., Holenstein, F., Sterne, J., Bartlett, C., & Egger, M. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.115 · Cited by 892
Epidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891
The Medical Review Article: State of the Science
MULROW, C. D. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-106-3-485 · Cited by 856
Users' Guides to the Medical Literature: IX. A Method for Grading Health Care Recommendations
Guyatt, G. H. (1995) · JAMA · doi:10.1001/jama.1995.03530220066035 · Cited by 846
A unification of models for meta-analysis of diagnostic accuracy studies
Harbord, R. M., Deeks, J. J., Egger, M., Whiting, P., & Sterne, J. A. C. (2007) · Biostatistics · doi:10.1093/biostatistics/kxl004 · Cited by 751
Better reporting of randomised controlled trials: the CONSORT statement
Altman, D. G. (1996) · BMJ · doi:10.1136/bmj.313.7057.570 · Cited by 609
An evidence-based practice guideline for the peer review of electronic search strategies
Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.012 · Cited by 608
Lee, Y. J., Lee, J. M., Lee, J. S., Lee, H. Y., Park, B. H., Kim, Y. H., Han, J. K., & Choi, B. I. (2015) · Radiology · doi:10.1148/radiol.14140690 · Cited by 526
Meta-analytic methods for diagnostic test accuracy
Irwig, L., Macaskill, P., Glasziou, P., & Fahey, M. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00099-c · Cited by 467
Colorectal Cancer: CT Colonography and Colonoscopy for Detection—Systematic Review and Meta-Analysis
Pickhardt, P. J., Hassan, C., Halligan, S., & Marmo, R. (2011) · Radiology · doi:10.1148/radiol.11101887 · Cited by 467
Estimating Diagnostic Accuracy from Multiple Conflicting Reports: A New Meta-analytic Method
Littenberg, B., & Moses, L. E. (1993) · Medical Decision Making · doi:10.1177/0272989x9301300408 · Cited by 434
State‐of‐the‐Evidence Reviews: Advantages and Challenges of Including Grey Literature
Benzies, K. M., Premji, S., Hayden, K. A., & Serrett, K. (2006) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2006.00051.x · Cited by 408
No role for quality scores in systematic reviews of diagnostic accuracy studies
Whiting, P., Harbord, R., & Kleijnen, J. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-19 · Cited by 337
A Systematic Review and Meta-Analysis of Diagnostic Performance of Imaging in Acute Cholecystitis
Kiewiet, J. J. S., Leeuwenburgh, M. M. N., Bipat, S., Bossuyt, P. M. M., Stoker, J., & Boermeester, M. A. (2012) · Radiology · doi:10.1148/radiol.12111561 · Cited by 329
Self-citationYavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001308 · Cited by 293
Harbord, R. M., Whiting, P., Sterne, J. A. C., Egger, M., Deeks, J. J., Shang, A., & Bachmann, L. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.09.013 · Cited by 217
Macaskill, P. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.019 · Cited by 210
Pham, B., Klassen, T. P., Lawson, M. L., & Moher, D. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.08.021 · Cited by 206
Leeflang, M. M. G., Scholten, R. J. P. M., Rutjes, A. W. S., Reitsma, J. B., & Bossuyt, P. M. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.07.014 · Cited by 162
Self-citationOverinterpretation and Misreporting of Diagnostic Accuracy Studies: Evidence of “Spin”
Ochodo, E. A., de Haan, M. C., Reitsma, J. B., Hooft, L., Bossuyt, P. M., & Leeflang, M. M. G. (2013) · Radiology · doi:10.1148/radiol.12120527 · Cited by 158
Self-citationSystematic review of publication bias in studies on publication bias
Dubben, H.-H., & Beck-Bornholdt, H.-P. (2005) · BMJ · doi:10.1136/bmj.38478.497164.f7 · Cited by 153
McInnes, M. D. F., Kielar, A. Z., & Macdonald, D. B. (2011) · Radiology · doi:10.1148/radiol.11110333 · Cited by 150
Self-citationTunis, A. S., McInnes, M. D. F., Hanna, R., & Esmail, K. (2013) · Radiology · doi:10.1148/radiol.13130273 · Cited by 146
Self-citationAbandoning Daily Routine Chest Radiography in the Intensive Care Unit: Meta-Analysis
Oba, Y., & Zaza, T. (2010) · Radiology · doi:10.1148/radiol.10090946 · Cited by 137
Toor, S. S., Jaberi, A., Macdonald, D. B., McInnes, M. D. F., Schweitzer, M. E., & Rasuli, P. (2012) · American Journal of Roentgenology · doi:10.2214/ajr.11.8362 · Cited by 128
Self-citationWhiting, P. F., Westwood, M. E., Beynon, R., Burke, M., Sterne, J. A. C., & Glanville, J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.006 · Cited by 117
Evidence-based Radiology: Steps 1 and 2—Asking Answerable Questions and Searching for Evidence
Staunton, M. (2007) · Radiology · doi:10.1148/radiol.2421052135 · Cited by 95
CT in Adults: Systematic Review and Meta-Analysis of Interpretation Discrepancy Rates
Wu, M. Z., McInnes, M. D. F., Macdonald, D. B., Kielar, A. Z., & Duigenan, S. (2013) · Radiology · doi:10.1148/radiol.13131114 · Cited by 84
Self-citationOverman, V. P. (2007) · International Journal of Dental Hygiene · doi:10.1111/j.1601-5037.2007.00226.x · Cited by 74
Ebell, M. H., Siwek, J., Weiss, B. D., Woolf, S. H., Susman, J. L., Ewigman, B., & Bowman, M. A. (2004) · PubMed · Cited by 68
Overinterpretation of Clinical Applicability in Molecular Diagnostic Research
Lumbreras, B., Parker, L. A., Porta, M., Pollán, M., Ioannidis, J. P. A., & Hernández-Aguado, I. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.121517 · Cited by 65
van der Pol, C. B., McInnes, M. D. F., Petrcich, W., Tunis, A. S., & Hanna, R. (2015) · PLOS ONE · doi:10.1371/journal.pone.0119892 · Cited by 56
Self-citationSchieda, N., McInnes, M. D. F., & Cao, L. (2014) · European Radiology · doi:10.1007/s00330-014-3147-4 · Cited by 54
Self-citationWeeks, S. M. (2013) · Western Journal of Nursing Research · doi:10.1177/0193945913491839 · Cited by 52
Role of Radiologic Imaging in Irritable Bowel Syndrome: Evidence-based Review
O’Connor, O. J., McSweeney, S. E., McWilliams, S., O’Neill, S. B., Shanahan, F., Quigley, E. M. M., & Maher, M. M. (2011) · Radiology · doi:10.1148/radiol.11110423 · Cited by 45
Do published search filters to identify diagnostic test accuracy studies perform adequately?
Ritchie, G., Glanville, J., & Lefebvre, C. (2007) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2007.00735.x · Cited by 39
Spin in Radiology Research: Let the Data Speak for Themselves
Levine, D., Bankier, A. A., & Kressel, H. Y. (2013) · Radiology · doi:10.1148/radiol.13130108 · Cited by 5
Research works printed but not linked: 8
- Ebell MH, 2004, J Fam Pract, 53, 111
- Macaskill P, 2010, Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0. The Cochrane Collaboration
- Beynon R, 2013, Cochrane Database Syst Rev, 9
- Hopewell S, 2007, Cochrane Database Syst Rev
- Citrome L, 2014, Innov Clin Neurosci, 11, 42
- Bossuyt P, 2013, Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 0.9. The Cochrane Collaboration
- Taylor T, 2014, Cochrane Database Syst Rev, 3, CD009694
- Josephson CB, 2014, Cochrane Database Syst Rev, 9
Cited by
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
McInnes, M. D. F., Moher, D., Thombs, B. D., McGrath, T. A., Bossuyt, P. M., and the PRISMA-DTA Group, Clifford, T., Cohen, J. F., Deeks, J. J., Gatsonis, C., et al. (2018) · JAMA · doi:10.1001/jama.2017.19163 · Cited by 3,210
“Systematic reviews of diagnostic test accuracy synthesize data from primary studies to provide insight into the ability of medical tests to detect a target condition; they also can provide estimates of test performance, allow comparisons of the accuracy of different tests, and facilitate the identification of sources…”
InfluentialSelf-citationvan der Pol, C. B., Lim, C. S., Sirlin, C. B., McGrath, T. A., Salameh, J.-P., Bashir, M. R., Tang, A., Singal, A. G., Costa, A. F., Fowler, K., et al. (2019) · Gastroenterology · doi:10.1053/j.gastro.2018.11.020 · Cited by 334
Self-citationMaw, A. M., Hassanin, A., Ho, P. M., McInnes, M. D. F., Moss, A., Juarez-Colunga, E., Soni, N. J., Miglioranza, M. H., Platz, E., DeSanto, K., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0703 · Cited by 309
Self-citationMcGrath, T. A., Alabousi, M., Skidmore, B., Korevaar, D. A., Bossuyt, P. M. M., Moher, D., Thombs, B., & McInnes, M. D. F. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0590-8 · Cited by 142
“Thus, some items in the original PRISMA checklist may not apply to DTA reviews, and some essential items necessary for reporting DTA systematic reviews may be lacking [2, 6, 13, 14].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- McInnes, M. D. F., & Bossuyt, P. M. M.
- Affiliations
- University of Ottawa (CA)
- Venue
- Radiology
- Published
- 2015-10
- Volume
- 277
- Issue
- 1
- Pages
- 13-21
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2114832664
- Semantic Scholar
- 3a7a8f4cad83a1bb9a35d26da9ac6cb134fbbae8
- Cited by
- 99
- References
- 53
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 6 · 2023: 7 · 2022: 7 · 2021: 13 · 2020: 15 · 2019: 12 · 2018: 18 · 2017: 8 · 2016: 6
- FWCI
- 1.76
- Type
- journal article
- Language
- en
- Open access
- green
- pure.amsterdamumc.nl
- Licence
- other-oa
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- systematic reviews, meta-analysis, imaging research, diagnostic accuracy, review methodology, bias assessment, publication bias, heterogeneity, literature searching, review questions
- MeSH terms
- Diagnostic Imaging, Humans, Review Literature as Topic, Meta-Analysis as Topic, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.