The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,501–2,600 of 4,578
A pre-hospital mixed methods systematic review protocol
McManamny, T. E., Ortega, M., Munro, S., Jennings, P., & Whitley, G. A. (2023) · British Paramedic Journal · doi:10.29045/14784726.2023.9.8.2.38 · Cited by 4
Leiria, M., & Nunes, C. (2023) · The International Journal of Children s Rights · doi:10.1163/15718182-31030004 · Cited by 4
Yeboah, E. A., Adegboye, A. R. A., & Kneafsey, R. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-071965 · Cited by 4
Li, T., Zhang, Y., Yao, L., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1272738 · Cited by 3
Aderoba, A. K., Kabra, R., & Kiarie, J. N. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02356-5 · Cited by 4
Berger, D. (2023) · Engaged Management ReView · doi:10.28953/2375-8643.1128 · Cited by 10
Deng, J., Sarraf, L., Hotte‐Meunier, A., & Sauvé, G. (2023) · Journal of Vocational Rehabilitation · doi:10.3233/jvr-230045 · Cited by 4
“A quality appraisal of each included study was made subsequently following Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
Yeo, Y., Lau, Y., Ang, W. W., Wong, Q. F., Tam, H. L., Loh, W. L., Wong, S. H., & Lau, S. T. (2023) · Geriatric Nursing · doi:10.1016/j.gerinurse.2023.10.012 · Cited by 5
Performance of tunnelled, non‐cuffed central venous catheters in infants: A scoping review
Dean, A., Doyle, R., & Ullman, A. J. (2023) · Journal of Paediatrics and Child Health · doi:10.1111/jpc.16509 · Cited by 3
Krishnan, N., & Ireland, J. L. (2023) · Journal of Forensic Psychology Research and Practice · doi:10.1080/24732850.2023.2281431 · Cited by 4
“…studies (Thomas et al., 2004); the Critical Appraisal Skills Programme (CASP) qualitative checklist (QC) was used to appraise all qualitative studies (CASP Critical Appraisal Skills Programme, 2018) and The Mixed Methods Appraisal Tool (MMAT) to appraise mixed method studies (Hong et al., 2018).”
Berger‐Höger, B., Lewis, K. B., Cherry, K., Finderup, J., Gunderson, J., Kaden, J., Kienlin, S. M., Rahn, A. C., Sikora, L., Stacey, D., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-071478 · Cited by 6
“38 The MMAT is used to appraise the methodological quality of studies with diverse designs (qualitative, quantitative and mixed-methods research) included in systematic mixed studies reviews.”
MethodEDUCATIONAL INTERVENTION WITH NURSES REGARDING CHILDCARE CONSULTATION: A MIXED METHOD STUDY
Vieira, D. D. S., Soares, A. R., Guedes, A. T. A., et al. (2023) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-0132en · Cited by 3
A systematic review of research on educational superintendents
Holmqvist, M., & Ekström, M. L. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2307142 · Cited by 11
“Finally, after a quality check using the mixed-methods appraisal tool (MMAT, Bartlett et al., 2018), which involved a careful reading of the full text of each article, 28 articles were excluded in the first round, and two more articles were excluded in the final reading because they did not satisfy…”
Ahmed, B. H., Voss, J. G., Schiltz, N. K., Naif, A. A., Ruksakulpiwat, S., & Griggs, S. A. (2024) · Clinical Nursing Research · doi:10.1177/10547738231223577 · Cited by 4
“Quality was appraised using the Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong, 2018).”
MethodAlexander, E. D., Chung, V.-H.-A., Yacovelli, A., Sarmiento, I., & Andersson, N. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-078193 · Cited by 10
“We will use the Mixed Methods Appraisal Tool (MMAT 2018) 31 to report the overall quality of the included studies and their implications for the interpretation of our results.”
MethodInformation needs for GPs on type 2 diabetes in Western countries: a systematic review
Andersen, T. H., Marcussen, T. M., & Nørgaard, O. (2024) · British Journal of General Practice · doi:10.3399/bjgp.2023.0531 · Cited by 4
Minian, N., Wong, M., Hafuth, S., Rodak, T., Rahimi, A., Gjomema, D., Rose, J., Zawertailo, L. A., Ratto, M., & Selby, P. L. (2024) · BMC Public Health · doi:10.1186/s12889-024-18139-z · Cited by 6
“N a u s e a [ 31 , 39 , 72 , 74 – 76 , 44 – 46 , 48 – 50 , 80 , 81 , 51 , 88 , 90 , 56 , 59 , 62 , 64 , 85 , 61 ] , h e a d a c h e [ 39 , 72 , 75 , 46 , 80 , 88 , 62 , 64 ] , v o m i t i n g [ 35 , 42 , 76 , 45 , 46 , 64 ] , i n s o m n i a / s l ee p p r o b l e m [ 35 , 38 , 72 , 75 , 46 , 53 ,…”
Process Evaluations of Diabetes Self-Management Programs: A Systematic Review of the Literature
Nsobundu, C., Nmadu, Y. W., Wagle, N. S., Foster, M., McKyer, E. L. J., Sherman, L. D., Ory, M. G., & Burdine, J. N. (2024) · American Journal of Health Promotion · doi:10.1177/08901171241238554 · Cited by 3
“20,21 Based on MMAT ’ s classi fi cation of empirical studies, the studies included in this review composed of more than 1 article were categorized as a combination of the research designs of their separate articles (i.e., 1 RCT +1 QUAL-type study).”
Yari, A., Hasan, M. K., Khoshsabegheh, H. Y., Boubakran, M. S., & Motlagh, M. E. (2024) · Disaster Medicine and Public Health Preparedness · doi:10.1017/dmp.2024.46 · Cited by 5
Dannhauser, F. C., Taylor, L. C., Tung, J., & Usher‐Smith, J. A. (2024) · Journal of Community Genetics · doi:10.1007/s12687-024-00709-8 · Cited by 4
“Two reviewers (FD and LT) independently completed quality assessment for all eligible papers using the Mixed Meth-ods Appraisal Tool (MMAT) (Hong et al. 2018), which is appropriate for assessing the quality of qualitative, quantitative and mixed-methods research.”
Tetteh, C. D., Ncayiyana, J. R., Makhunga, S., Manyeh, A. K., Asiamah, E. A., & Ginindza, T. G. (2024) · Southern African Journal of Infectious Diseases · doi:10.4102/sajid.v39i1.553 · Cited by 5
“The Mixed Method Appraisal Tool (MMAT) 49 will be used to appraise the articles selected for the study by the two screeners.”
MethodSenyange, B., Steur, H. D., Wesana, J., Gellynck, X., & Huylenbroeck, G. V. (2024) · The Journal of Horticultural Science and Biotechnology · doi:10.1080/14620316.2024.2376591 · Cited by 5
“In contrast to other tools, MMAT focuses on the central methodological criteria and assesses studies in five categories: qualitative, randomised, non-randomised, descriptive and mixed methods (Hong et al., 2018).”
InfluentialMullen, L., Evans, M., & Baillie, L. (2024) · European Journal of Special Needs Education · doi:10.1080/08856257.2024.2372966 · Cited by 5
Stepanova, V., Poppleton, A., & Ponsford, R. (2024) · Health Expectations · doi:10.1111/hex.14155 · Cited by 4
Multiple sclerosis and identity: a mixed-methods systematic review
Graziano, F., Calandri, E., Borghi, M., Giacoppo, I., Verdiglione, J., & Bonino, S. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2024.2392039 · Cited by 7
“The overall quality of the selected qualitative studies was good according to the MMAT criteria [22] (Supplementary Appendix 3).”
InfluentialValarezo-Bravo, O. F., Guzmán‐González, M., & Garrido-Rojas, L. (2024) · Revista Latinoamericana de Psicología · doi:10.14349/rlp.2024.v56.11 · Cited by 3
Badrakh, A., Buglass, S. L., Betts, L. R., & Abell, L. (2024) · Current Psychology · doi:10.1007/s12144-024-06698-8 · Cited by 7
“Likewise, for mixed-method studies, the Mixed Methods Appraisal Tool (MMAT; (Hong et al., 2018) was used.”
Experiences of internationally educated nurses working with older adults: A scoping review protocol
Kokorelias, K. M., Saragosa, M., Abdelhalim, R., Philippopoulos, E., & Vo, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307795 · Cited by 3
Kelley, M. L., Strowger, M., Gabelmann, J. M., Vasic, S., Rivera, I. E., Fleming, R., Burgin, E. E., Bravo, A. J., Fleming, W. H., Gaylord, S. A., et al. (2024) · Counseling and Values · doi:10.1163/2161007x-bja10011 · Cited by 4
Threlfall, J. M., Jobling, H., & Graham, K. (2024) · British Journal of Learning Disabilities · doi:10.1111/bld.12628 · Cited by 3
Thornton, M., Blamires, J., Foster, M., Mowat, R. M., & Haven, S. (2024) · Nurse Education in Practice · doi:10.1016/j.nepr.2024.104227 · Cited by 5
Vallury, K. D., Asher, A., Sarri, O., et al. (2025) · Sexual Health · doi:10.1071/sh24170 · Cited by 3
Chan, S. H., Cheston, R., Steward-Anderson, C., Yu, C. H., Dodd, E., & Coulthard, E. J. (2025) · Healthcare · doi:10.3390/healthcare13030296 · Cited by 3
Kalitsilo, L., Abdullahi, L. H., Mbeye, N. M., Mwandira, L., Hara, H., Mitambo, C., & Oronje, R. N. (2025) · PLoS ONE · doi:10.1371/journal.pone.0302279 · Cited by 7
Brennan, E., Clarke, M. (., & Guérin, S. (2025) · Palliative & Supportive Care · doi:10.1017/s1478951524001743 · Cited by 4
“At the same time, quality appraisal of the articles was conducted using the Mixed Methods Appraisal Tool (Hong et al. 2018).”
Puente-López, E., Pina, D., Shura, R. D., López-López, R., Merten, T., & Martínez‐Jarreta, B. (2025) · The Clinical Neuropsychologist · doi:10.1080/13854046.2025.2469354 · Cited by 3
“…Tool (RoB 2; Sterne et al., 2019), the Cochrane Risk of Bias in non-Randomized Studies of Interventions (RoBInS-I Tool; Sterne et al., 2016), the Mixed Methods Appraisal Tool (MMAT; hong et al., 2018), and the critical appraisal tools of the Joanna Briggs Institute (JBI; Aromataris & Munn, 2020).”
Rail suicide: A systematic review using systems thinking
Read, G. J. M., Elstak, I., Naweed, A., Cox, J. A., Too, L. S., Hermens, D. F., McClure, R., & Salmon, P. M. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106856 · Cited by 4
Petrides, M., Peletidi, A., Nena, E., Constantinidis, T. C., & Kontogiorgis, C. A. (2025) · Journal of Pharmaceutical Policy and Practice · doi:10.1080/20523211.2025.2487046 · Cited by 3
“The review included studies from various locations and employed the validated Mixed Methods Appraisal Tool (MMAT), version 2018, to evaluate the quality of the selected studies.”
InfluentialEquity and Access in Digital Transformation: Enhancing Curriculum Delivery at Rural Universities
Ajani, O. A. (2025) · E-Journal of Humanities Arts and Social Sciences · doi:10.38159/ehass.20256515 · Cited by 4
Empowering patient choice: a systematic review of decision aids for benign prostatic hyperplasia
Hollands, C., Gray, D., Marren, A., Hindley, R. G., & Husted, M. (2025) · British Journal of Urology · doi:10.1111/bju.16797 · Cited by 3
“Risk-of-bias evaluation using the MMAT [40] focused on appraising each included article based on two screening questions, followed by fi ve core criteria that coincided with the article ’ s methodological category.”
InfluentialProcesses of co-response crisis mental health models: a scoping review
Eaton, A. D., Rowe, M. W., Nguyen, A., Lowe, L., Fletcher, K., Adenwala, A., Sudom-Young, S., Rackow, R., Poellet, L., McDonald, M., et al. (2025) · Advances in Mental Health · doi:10.1080/18387357.2025.2522434 · Cited by 6
Kostromitina, M., Naismith, B., Burstein, J. C., & Plonsky, L. (2025) · Review of Education · doi:10.1002/rev3.70089 · Cited by 5
Kelleher, A. B., O’Donovan, M. R., O’Doherty, D., Lavery, R., Lehane, E. A., & Saab, M. M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0332031 · Cited by 3
“One researcher used the RCT items from the Mixed Method Appraisal Tool (MMAT) [32] to appraise the methodological quality for each study.”
Nweke, M., Mshunqane, N., Govender, N., & Akinpelu, O. F. (2021) · African Journal for Physical Activity and Health Sciences (AJPHES) · doi:10.37597/ajphes.2021.27.1.8 · Cited by 6
Strategies in promoting creative thinking skills in science classroom: A systematic review
Nasir, W. M. F. W. M., Halim, L., & Arsad, N. M. (2022) · Cypriot Journal of Educational Sciences · doi:10.18844/cjes.v17i12.7605 · Cited by 3
“The reviewers were directed by these five criteria for their quality assessment, which they adapted from the guidelines provided by Hong et al. (2018): 1) Are teaching strategies for promoting creative thinking skills among the articles' primary objectives?”
MethodErku, D., Khatri, R. B., Endalamaw, A., Wolka, E., Nigatu, F., Zewdie, A., & Assefa, Y. A. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-070370 · Cited by 4
Systematic Review of Mediatization of Politics in Southeast Asia
Malaysia, U. K., Kho, S. N., Zawawi, J. W. M., & Malaysia, U. P. (2023) · Jurnal Komunikasi Malaysian Journal of Communication · doi:10.17576/jkmjc-2023-3901-17 · Cited by 8
Choi, J., Son, Y., & Lee, K. (2020) · Journal of Korean Critical Care Nursing · doi:10.34250/jkccn.2020.13.1.27 · Cited by 3
“본 연구에 양적연구, 질적연구 및 혼합연구가 혼재되어 질 평가를 위해 Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) 기준으로 사용하였다.”
Use of PADIS Assessment Tools by Critical Care Nurses: An Integrative Review
Waterfield, D., & Barnason, S. A. (2020) · Western Journal of Nursing Research · doi:10.1177/0193945920973025 · Cited by 3
Cave, L., Milnes, L. J., & McHugh, G. A. (2021) · Journal of Child Health Care · doi:10.1177/13674935211029124 · Cited by 3
“The methodological quality of included studies was critically appraised using the Mixed Methods Appraisal Tool (MMAT) (version 2018) (Hong et al., 2018).”
BackgroundMethodEssential skills for using research evidence in public health policy: a systematic review
Ziam, S., Gignac, P., Courant, É., & Sween-Cadieux, E. M. (2021) · Evidence & Policy · doi:10.1332/174426421x16250726996691 · Cited by 3
Sa’id, I. I. M., Nadal, I. P., Forbes, A., Goldsmith, K., Ismail, I. Z., Hassan, F., Ching, S. M., Guess, N. D., Murphy, H. R., Prina, M. A., et al. (2021) · International Journal of Qualitative Methods · doi:10.1177/16094069211034010 · Cited by 4
“Hence, a unique tool known as Mixed Method Appraisal Tool by Hong et al. (2018) will be used to assess the quality of different study designs (see Figure 3).”
BackgroundMethodDwyer, L., Dowding, D., & Kearney, R. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055587 · Cited by 3
Leaune, É., Leclerc, J. B., Fender, R., Notredame, C.-É., Jurek, L., & Poulet, E. (2022) · International Journal of Mental Health · doi:10.1080/00207411.2022.2064176 · Cited by 3
“The researchers did not assign the overall quality score as it is discouraged by Hong et al. (2018), but the methodological quality of the studies was assessed in accordance to the - guideline provided by Hong et al. (2018).”
BackgroundMethodInfluentialMoisture accumulation detection technologies for identifying pressure injuries: a literature review
Bone, M., Latimer, S. L., Walker, R., & Gillespie, B. M. (2022) · Wound Practice and Research · doi:10.33235/wpr.30.4.207-215 · Cited by 3
“Therefore, to improve the objectivity in assessment and inter-rater reliability of the MMAT checklist, two researchers independently rated the studies (MB, SL).”
InfluentialKilinç, N., Yılmaz, M., & Türk, Ş. Ş. (2023) · Journal of Urban Planning and Development · doi:10.1061/jupddm.upeng-4489 · Cited by 3
Esmaeili, M., Abdi, F., Shafiee, G., Rastad, H., Asayesh, H., Abdar, Z. E., Baygi, F., & Qorbani, M. (2020) · Research Square · doi:10.21203/rs.3.rs-44648/v1 · Cited by 3
Budzynski‐Seymour, E., Turvey, S., Paterson, J., Jones, M. A., & Steele, J. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/kh37d · Cited by 3
Effective Virtual Interventions to Enhance Psychological Capital: A mixed-methods systematic reviewPreprint
Xu, J. H., Qureshi, A. R., Dabagh, Y. A., Kin, C. L., & Khan, R. (2021) · doi:10.31234/osf.io/dpjuy · Cited by 3
Hinds, J. M., Parkhouse, T., & Hotchin, V. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4g8ec · Cited by 5
Cadogan, E., Murphy, M. F., Lionetti, F., & Setti, A. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/edy7k · Cited by 5
Mental health, mobility and climate change: a scoping review protocol v1Preprint
Goudet, J.-M., Arif, F. B., Ahmed, H. U., & Ridde, V. (2022) · doi:10.17504/protocols.io.5qpvor9dbv4o/v1 · Cited by 2
Gilliland, A. R., Gaughan, E. E., Meek, H., Biyani, C. S., Ijaz, F., Gabriel, G., Mathew, R. K., & Mushtaq, F. (2024) · medRxiv · doi:10.1101/2024.06.17.24309027 · Cited by 5
Ajibade, B. O., Ubah, P., Kainesie, J., Suleiman, U., Omoruyi, M., & Olubunmi, A.-I. T. (2024) · medRxiv · doi:10.1101/2024.06.24.24309436 · Cited by 4
Alotaibi, N., Wilson, C. B., & Traynor, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4816097/v1 · Cited by 3
Locke, A., James, M., Jones, H. E., Davies, R., Williams, F., & Brophy, S. T. (2024) · medRxiv · doi:10.1101/2024.09.03.24312981 · Cited by 3
Brothers, T. D., Lewer, D., Bonn, M., Kim, I., Comeau, E. M., Figgatt, M. C., Eger, W. H., Webster, D., Hayward, A. C., & Harris, M. (2024) · medRxiv · doi:10.1101/2024.09.20.24313898 · Cited by 6
Choudhury, S., Medina-Lara, A., Bi, A. Z., Ricarte, P., Morrish, N., & Veettil, P. C. (2026) · Foods · doi:10.3390/foods15081381 · Cited by 3
Taheri, S., Sichani, M. G., & Shabani, A. (2021) · Haft Hesar Journal of Environmental Studies · doi:10.52547/hafthesar.10.36.9 · Cited by 3
Bushell, D., Jones, C., & Moro, C. (2023) · Frontiers in Dementia · doi:10.3389/frdem.2023.1156863 · Cited by 6
“The use of defined inclusion and exclusion criteria, rigorous search strategy of eight databases and validated MMAT2018 use for quality assessment are strengths of this review.”
MethodInfluentialHormazábal‐Salgado, R., Osman, A. D., Troncoso, M. P., Whitehead, D., & Hills, D. (2023) · Journal of Social Work in End-of-Life & Palliative Care · doi:10.1080/15524256.2023.2229041 · Cited by 3
“…Institute, 2022), the Study Quality Assessment Tools from the National Heart Lung and Blood Institute (2021) for Controlled Intervention Studies, Case-Control studies, and Observational Cohort and Cross-Sectional Studies in addition to the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
MethodMukona, D. M., George, R., James, J., Joseph, S., Mukasa, J. P., Timberlake, C., Monaco, B., Abdulla, S., Khalaf, A., & Burnett, E. (2024) · Journal of Infection Prevention · doi:10.1177/17571774241236247 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT) will be used to assess the methodological quality of the included mixed method studies (Hong et al., 2018).”
Jäger, P., Hirt, J., Nordhausen, T., Vetsch, J., Balzer, K., Dichter, M., Müller, G., Schley, A., & Neyer, S. (2024) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2024.01.006 · Cited by 2
Impacts of duty belts and load placement on police officers: A systematic review
Berner, N., Biilmann, C., Hunter, D., Canetti, E. F. D., Schram, B. L., Dawes, J. J., Lockie, R. G., & Orr, R. M. (2024) · Nauka bezbednost policija · doi:10.5937/nabepo29-49879 · Cited by 5
Non-ictal, interictal and ictal déjà vu: a systematic review and meta-analysis
Hadzic, A., & Andersson, S. (2024) · Frontiers in Neurology · doi:10.3389/fneur.2024.1406889 · Cited by 5
“The study risk of bias assessment, using the MMAT version 18 (23, 24), showed that out of the 46 studies evaluated, 46 studies (100”
MethodMo, Y., McLeod, S., McCormack, J. C., Leong, S. Y., Conner, T. S., Oey, I., & Peng, M. (2024) · Sustainability · doi:10.3390/su16146241 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT) [43] was used to determine the risk of bias in the included studies, and to assess both quality and risk of bias within five domains: participants, measurements, outcome data, confounding factors, and administration of the exposure.”
Relapse prevention following guided self-help for common health problems: A Scoping Review
Nawaz, S., Bee, P., Devaney, H., & Faija, C. (2024) · Cognitive Therapy and Research · doi:10.1007/s10608-024-10520-x · Cited by 3
“This tool was chosen as it allows the appraisal of empirical studies across different study designs including qualitative, quantitative randomised controlled trials, quantitative non-randomised controlled trials, quantitative descriptive and mixed meth-ods (Hong et al., 2018).”
Malden, S., McDougall, C., Nelson, C. W., Carroll-Monteil, E., & Jepson, R. (2024) · Active Travel Studies · doi:10.16997/ats.1510 · Cited by 5
Xie, Y., Fadahunsi, K. P., Flynn, P., Taylor-Robinson, S. D., Gallagher, J., Cullen, W., & O’Donoghue, J. (2024) · JMIR Research Protocols · doi:10.2196/59927 · Cited by 2
Cheong, D., Pereira, T. L.-B., Huang, Z., & Shorey, S. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17472 · Cited by 3
Campos, L. N., Bryce‐Alberti, M., Gerk, A., Hill, S. K., Calderon, C., Zaigham, M., del Valle, D. D., Mita, C., Juran, S., Ferreira, J., et al. (2024) · The Lancet Regional Health - Americas · doi:10.1016/j.lana.2024.100908 · Cited by 2
“23 The primary reason for selecting this tool was its demonstrated effectiveness in studies with heterogeneous methodologies.”
Baradaran, A., Tolentino, R., Grad, R. M., Ganache, I., Gore, G. C., Rahimi, S. A., & Pluye, P. (2024) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462324000370 · Cited by 3
“The majority (82.5 percent) of included studies had an MMAT score of 80 percent or higher.”
InfluentialSelf-citationPhilbin, S. E., Harris, A., Balbale, S. N., Bilaver, L., Beestrum, M., & McHugh, M. (2024) · Journal of Substance Use and Addiction Treatment · doi:10.1016/j.josat.2024.209600 · Cited by 2
Coping After Violent Loss: A Systematic Literature Review and Organizing Model
Rice, A. J., Fisher, J. E., & Cozza, S. J. (2024) · Journal of Loss and Trauma · doi:10.1080/15325024.2024.2415706 · Cited by 2
Exploring Australian nursing and midwifery informatics roles: A scoping review
Jedwab, R. M., Langtree, T., LoPresti, K. C., Jones, A. J., & Dobroff, N. K. (2025) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2025.01.002 · Cited by 2
Iwuchukwu, G., Anttila, M., Lantta, T., Varpula, J., & Ameel, M. (2025) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001263 · Cited by 5
Tyldesley‐Marshall, N., Johnson, R., Parr, J., Brown, A., Ghosh, I., Mehrabian, A., Chen, Y., & Grove, A. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1513668 · Cited by 4
Chacko, J., Mazza, K., Stathakos, K., Kim, D., & Carlo, L. B. (2025) · Cureus · doi:10.7759/cureus.79467 · Cited by 4
“To assure the quality of the review and avoid the risk of bias, reviewer 1 (JC) and reviewer 2 (LC or KC) independently used the Mixed Methods Appraisal Tool (MMAT) version 2018 [19].”
InfluentialBoonyananth, N., & Swords, L. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251320979 · Cited by 6
“The Mixed-Methods Appraisal Tool (MMAT; Hong et al., 2018) was also used to assess the quality of each study included.”
Transition from hospital to home care: a mixed methods study in light of Meleis’s Theory
Gama, L. M. D. P., Ribeiro, K. R. C., de Oliveira, J. L. C., et al. (2025) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2023-0357 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT) was used to direct the study construction and guide the methodology (12) .”
Adeniji, O. A., Stenner, K. L., Pappas, E., Traynor, V., Chrysanthaki, T., & Carey, N. (2025) · Global Implementation Research and Applications · doi:10.1007/s43477-025-00155-9 · Cited by 6
“As the review included studies with quantitative, qualitative, and mixed-methods research, the mixed methods appraisal tool (MMAT) (2018) (Hong et al., 2018) was used to review quality.”
Jones, R., Jackson, D., Rice, K., Usher, K. J., Davies, R., Toole‐Anstey, C., Chhabra, J., Smith, J. A., Morley, L., Russ, E., et al. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251320988 · Cited by 4
“The two reviewers independently reviewed the included articles using Mixed Methods Assessment Tool (MMAT; (Hong et al., 2018; see Supplemental File), the results were compared before finalizing their assessment.”
Nurse prescribing in China: A systematic review
Li, C., & Lee, G. (2025) · The Asia Pacific Scholar · doi:10.29060/taps.2025-10-2/ra3261 · Cited by 4
O’Keefe, B., Yuen, E. Y. N., Perlen, S., Dow, B., & Hutchinson, A. M. (2025) · Australian Journal on Ageing · doi:10.1111/ajag.70017 · Cited by 2
“The authors used the Mixed Methods Appraisal Tool (MMAT) version 2018, 37 to assess the methodological quality of the included articles.”
Cooke, K., Ridgway, K., Pecora, L. A., Westrupp, E. M., Hedley, D., Hooley, M., & Stokes, M. A. (2025) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-025-00498-x · Cited by 8
Shorey, S., Wong, J. C. M., Lim, D., Chua, C. M. S., Teng, J. Y., & Lim, L. H. K. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70133 · Cited by 3
“As recommended by Pluye and Hong, 18 none of the studies were scored or excluded from this review based on the MMAT appraisal.”
Board Game for Children Coping with Cancer in Brazil: A Mixed Methods Study
Amador, D. D., Alves, D. F. D. S., & Mandetta, M. A. (2025) · Aquichan · doi:10.5294/aqui.2025.25.1.4 · Cited by 2
“…(when the results of a method are complemented with another method) and simultaneous procedures (seeking the possibilities of converging information to answer the study questions) (15, 16), according to the criteria established in the 2018 version of the Mixed Methods Appraisal Tool (16).”
Hualparuca‐Olivera, L., Caycho‐Rodríguez, T., Torales, J. C., Ramos‐Vera, C., Vigo-Ayasta, E., & Ventriglio, A. (2025) · Psychological Reports · doi:10.1177/00332941251335596 · Cited by 3
van der Spek, L., Breunis, L. J., Schayck, T. S.-V., Bauld, L., Ista, E., & Been, J. V. (2025) · Tobacco Control · doi:10.1136/tc-2024-059198 · Cited by 7
“47 In line with the recommendations of the MMAT developers, we present details alongside quality scores, considering studies that meet at least 60% of MMAT criteria as having good methodological quality.”
Pedersen, M. L., Bricca, A., Baker, J. A., Schjerning, O., Munk‐Olsen, T., & Gildberg, F. A. (2025) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2025.04.014 · Cited by 2
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.