The DOI matches its Crossref record; 39 of the 40 references it cites were found in a registry or an index.
Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool ( MMAT )
Open accessAbstract
Rationale, aims, and objectives: Systematic reviews combining qualitative, quantitative, and/or mixed methods studies are increasingly popular because of their potential for addressing complex interventions and phenomena, specifically for assessing and improving clinical practice. A major challenge encountered with this type of review is the appraisal of the quality of individual studies given the heterogeneity of the study designs. The Mixed Methods Appraisal Tool (MMAT) was developed to help overcome this challenge. The aim of this study was to explore the usefulness of the MMAT by seeking the views and experiences of researchers who have used it. Methods: We conducted a qualitative descriptive study using semistructured interviews with MMAT users. A purposeful sample was drawn from the researchers who had previously contacted the developer of the MMAT, and those who have published a systematic review for which they had used the MMAT. All interviews were transcribed verbatim and analyzed by 2 coders using thematic analysis. Results: Twenty participants from 8 countries were interviewed. Thirteen themes were identified and grouped into the 2 dimensions of usefulness, ie, utility and usability. The themes related to utility concerned the coverage, completeness, flexibility, and other utilities of the tool. Those regarding usability were related to the learnability, efficiency, satisfaction, and errors that could be made due to difficulties understanding or selecting the items to appraise. Conclusions: On the basis of the results of this study, we make several recommendations for improving the MMAT. This will contribute to greater usefulness of the MMAT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Ultimately, 84 studies with MMAT(Hong et al., 2018) [60] scores ≥ 3 were retained for analysis (Figure 3).”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [60]
The cited source says what the claim attributes to it.
In the citing paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“To rigorously appraise the quality of the included studies, we applied the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018), which is suitable for evaluating qualitative, quantitative, and mixed-methods research.”
From Digital Transformation in International Trade: Opportunities, Challenges, and Policy Implications (2025) · cites this paper as Hong et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Fereday, J., & Muir-Cochrane, E. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500107 · Cited by 11,829
The integrative review: updated methodology
Whittemore, R., & Knafl, K. (2005) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2005.03621.x · Cited by 10,241
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,552
What's in a name? Qualitative description revisited
Sandelowski, M. (2010) · Research in Nursing & Health · doi:10.1002/nur.20362 · Cited by 4,975
What is an adequate sample size? Operationalising data saturation for theory-based interview studies
Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., & Grimshaw, J. M. (2010) · Psychology & Health · doi:10.1080/08870440903194015 · Cited by 3,429
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Qualitative description – the poor cousin of health research?
Neergaard, M. A., Olesen, F., Andersen, R. S., & Søndergaard, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-52 · Cited by 2,420
Content validity in psychological assessment: A functional approach to concepts and methods.
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995) · Psychological Assessment · doi:10.1037//1040-3590.7.3.238 · Cited by 2,387
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Pluye, P., & Hong, Q. N. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182440 · Cited by 1,363 · cited 1×
“Systematic reviews combining qualitative, quantitative, and/or mixed methods studies are more and more popular because of their potential for addressing complex evaluation questions that matter in clinical practice.(1,2) Indeed, including different types of studies in a review can provide a richer understanding of the…”
BackgroundSelf-citationPace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 873
Hong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754 · cited 1×
“The first label refers to combining qualitative, quantitative, and/or mixed methods studies, while the second one can also refer to combining qualitative and quantitative methods (such as thematic synthesis and meta‐analysis).(7) Hereinafter, we will use the term “systematic mixed studies reviews” to designate this ty…”
MethodSelf-citationWhat Is Ecological Validity? A Dimensional Analysis
Schmuckler, M. A. (2001) · Infancy · doi:10.1207/s15327078in0204_02 · Cited by 638
Simera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-24 · Cited by 590
A Dictionary of Epidemiology (DRAFT)
(2014) · doi:10.1093/acref/9780199976720.001.0001 · Cited by 530
Introduction to systematic reviews
O’Connor, E., Whitlock, E. P., & Spring, B. J. (2011) · PsycEXTRA Dataset · doi:10.1037/e657462011-001 · Cited by 464
A systematic review of the content of critical appraisal tools
Katrak, P., Bialocerkowski, A. E., Massy‐Westropp, N., Kumar, S., & Grimmer, K. A. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-22 · Cited by 389
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375
Self-citationSandelowski, M., & Barroso, J. V. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100107 · Cited by 363
Mixed methods research synthesis: definition, framework, and potential
Heyvaert, M., Maes, B., & Onghena, P. (2011) · Quality & Quantity · doi:10.1007/s11135-011-9538-6 · Cited by 311
A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 166
Santiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.007 · Cited by 146
A proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121
Mhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.016 · Cited by 102
External Validity and Model Validity: A Conceptual Approach for Systematic Review Methodology
Khorsan, R., & Crawford, C. C. (2014) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2014/694804 · Cited by 100
Seo, H.-J., Kim, S. Y., Lee, Y. J., Jang, B.-H., Park, J.-E., Sheen, S.-S., & Hahn, S. K. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.013 · Cited by 74
Hartling, L., Bond, K., Santaguida, P. L., Viswanathan, M., & Dryden, D. M. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.010 · Cited by 69
How to Appraise the Studies: An Introduction to Assessing Study Quality
Petticrew, M. P., & Roberts, H. M. (2006) · Wiley eBooks · doi:10.1002/9780470754887.ch5 · Cited by 65
Barker, J. (2013) · SAGE Publications Ltd eBooks · doi:10.4135/9781529681413.n7 · Cited by 26
Verhage, A., & Boels, D. (2016) · Quality & Quantity · doi:10.1007/s11135-016-0345-y · Cited by 24
Including systematic reviews in PhD programmes and candidatures in nursing – ‘Hobson's choice’?
Olsson, C., Ringnér, A., & Borglin, G. (2014) · Nurse Education in Practice · doi:10.1016/j.nepr.2014.01.005 · Cited by 18
Heffernan, E., Oving, I., Barry, T., Phung, V.-H., Siriwardena, A., & Masterson, S. (2019) · HRB Open Research · doi:10.12688/hrbopenres.12969.2 · Cited by 7
Risk of bias assessment should not go beyond reporting assessment
Faggion, C. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.11.014 · Cited by 5
Flagged first, then by citations
1 of the 40 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Dixon‐Woods M, 2004, Integrative approaches to qualitative and quantitative evidence
- Heyvaert M, 2016, Using Mixed Methods Research Synthesis for Literature Reviews: The Mixed Methods Research Synthesis Approach
- Bai A, 2012, Quality Assessment Tools Project Report
- Pluye P, Proposal: A mixed methods appraisal tool for systematic mixed studies reviews
- 10.1037/1040-3590.7.3.238 · DOI · Paper page
Cited by
Sorted by citations
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationImproving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.008 · Cited by 1,082
“In addition, in interviews conducted with MMAT users to explore their views and experiences of the MMAT, concerns were raised about whether the tool included enough criteria to judge the quality of studies and criteria that were difficult to judge, in particular the criteria for qualitative and mixed methods studies […”
MethodSelf-citationMethodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2020) · JBI Evidence Synthesis · doi:10.11124/jbisrir-d-19-00169 · Cited by 747
Noyes, J., Booth, A., Moore, G., Flemming, K., Tunçalp, Ö., & Shakibazadeh, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000893 · Cited by 497
Ressel, M., Thompson, B., Poulin, M.-H., Normand, C. L., Fisher, M. H., Couture, G., & Iarocci, G. (2020) · Autism · doi:10.1177/1362361320910963 · Cited by 394
Metadata
- Authors
- Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P.
- Affiliations
- McGill University (CA)
- Department of Family Medicine McGill University Montréal QC Canada
- Corresponding author
- Pluye, Pierre
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2018-02-21
- Volume
- 24
- Issue
- 3
- Pages
- 459-467
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2794014169
- Cited by
- 1,107
- References
- 40
- Citations by year
- 2026: 227 · 2025: 230 · 2024: 176 · 2023: 156 · 2022: 130 · 2021: 78 · 2020: 84 · 2019: 20 · 2018: 5
- FWCI
- 34.23
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- Journal of Evaluation in Clinical Practice (publishedVersion)
- PubMed (publishedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- Funders
- Canadian Institutes of Health Research (301011)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- Mixed Methods Appraisal Tool, mixed methods, study quality appraisal, systematic reviews, tool usability, researcher experience, complex interventions
- MeSH terms
- Female, Humans, Interviews as Topic, Male, Research Design, Qualitative Research, Internationality, Quality Improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
