A Conceptual Framework for Critical Appraisal in Systematic Mixed Studies Reviews
Open accessAbstract
The past decade has been rich with methodological advancements in systematic reviews, several of which were inspired by the literature on mixed methods research. Systematic mixed studies reviews—that is, reviews combining qualitative and quantitative evidence—are increasingly popular as they can provide a better understanding of complex phenomena and interventions. However, they raise new challenges, especially regarding how to perform critical appraisal of the included studies that vary regarding the methodologies used. To address this challenge, conceptually clarifying critical appraisal is necessary. To this end, this article provides a framework for critical appraisal in systematic mixed studies reviews. This framework is an essential first step toward providing clear guidance on how to perform critical appraisal.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results of a study that are reported can be trustworthy.
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results of a study that are reported can be trustworthy (3, 22).”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [3]
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252 · see citing works
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Thomas, B. H., Ciliska, D., Dobbins, M., & Micucci, S. (2004) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1524-475x.2004.04006.x · Cited by 2,118 · see citing works
Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., Hsu, R., Katbamna, S., Olsen, R., Smith, L., et al. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-35 · Cited by 2,053 · see citing works
Building a Conceptual Framework: Philosophy, Definitions, and Procedure
Jabareen, Y. (2009) · International Journal of Qualitative Methods · doi:10.1177/160940690900800406 · Cited by 1,559 · see citing works
Pluye, P., & Hong, Q. N. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182440 · Cited by 1,363 · see citing works
Self-citationAppraising the Evidence: Reviewing Disparate Data Systematically
Hawker, S., Payne, S., Kerr, C., Hardey, M., & Powell, J. (2002) · Qualitative Health Research · doi:10.1177/1049732302238251 · Cited by 1,216 · see citing works
Hong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754 · cited 2× · see citing works
“In SMSRs, there are multiple synthesis method options (Hong et al., 2017).”
BackgroundMethodSelf-citationEvaluating meta-ethnography: systematic analysis and synthesis of qualitative research
Campbell, R., Pound, P., Morgan, M., Daker-White, G., Britten, N., Pill, R., Yardley, L., Pope, C., & Donovan, J. (2011) · Health Technology Assessment · doi:10.3310/hta15430 · Cited by 619 · see citing works
Integrating qualitative research with trials in systematic reviews
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., & Kavanagh, J. (2004) · BMJ · doi:10.1136/bmj.328.7446.1010 · Cited by 515 · see citing works
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · see citing works
Carroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235 · see citing works
Improving the reporting of public health intervention research: advancing TREND and CONSORT
Armstrong, R., Waters, E., Moore, L., Riggs, E., Cuervo, L. G., Lumbiganon, P., & Hawe, P. (2008) · Journal of Public Health · doi:10.1093/pubmed/fdm082 · Cited by 171 · see citing works
Frantzen, K. K., & Fetters, M. D. (2016) · Quality & Quantity · doi:10.1007/s11135-015-0261-6 · Cited by 91 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737 · see citing works
“This paper is the third of a three-part series on the topic of systematic reviews for information professionals (Hong & Pluye, 2018b; Pluye et al., 2018).”
Self-citationImproving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.008 · Cited by 1,082 · see citing works
“A conceptual framework on the quality appraisal in systematic mixed studies reviews was developed in which three dimensions of quality were presented: reporting, conceptual, and methodological [16].”
BackgroundSelf-citationThiyagarajan, A., Grant, C., Griffiths, F., & Atherton, H. (2020) · BJGP Open · doi:10.3399/bjgpopen20x101020 · Cited by 169 · see citing works
Moyes, E., Nutman, G., & Mirman, J. H. (2022) · Journal of Child & Adolescent Trauma · doi:10.1007/s40653-022-00454-5 · Cited by 19 · see citing works
- Authors
- Hong, Q. N., & Pluye, P.
- Affiliations
- McGill University (CA)
- Corresponding author
- Hong, Quan Nha
- Venue
- Journal of Mixed Methods Research
- Publisher
- SAGE Publications
- Published
- 2018-04-21
- Volume
- 13
- Issue
- 4
- Pages
- 446-460
- ISSN
- 1558-6898 (ISSN-L)
- 1558-6901
- OpenAlex
- W2799360482
- Cited by
- 139
- References
- 16
- Citations by year
- 2026: 19 · 2025: 26 · 2024: 17 · 2023: 15 · 2022: 17 · 2021: 20 · 2020: 14 · 2019: 7 · 2018: 3 · 2017: 1
- FWCI
- 8.39
- Type
- journal article
- Language
- en
- Open access
- Locations
- Journal of Mixed Methods Research (publishedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- Funders
- Canadian Institutes of Health Research (301011)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- critical appraisal, mixed methods, conceptual framework, methodological heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
