The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,701–2,800 of 4,578
Human impacts of flood disasters: A global systematic review (1980–2025)
Nooni, I. K., Sarpong, L., Kumi, N., Prempeh, N. A., Fianko, S. K., & Jin, Z. (2026) · H2Open Journal · doi:10.1016/j.htopen.2026.100019 · Cited by 8
Cação-Dias, B., Rothes, I., Carmo, C., Nunes, C., & Brás, M. S. V. (2026) · Archives of Suicide Research · doi:10.1080/13811118.2026.2666405 · Cited by 3
Arab, R. A. E., Moosa, O. A. A., Almagharbeh, W. T., Abuadas, F. H. A., Abdalla, N., Abdrbo, A. A., & Hassanein, S. E. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02400-6 · Cited by 6
“…synthesis of the primary evidence base; undertaking methodological and model-readiness appraisal using Mixed Methods Appraisal Tool (MMAT) [15, 16 ] and PROBAST + AI [ 17], where applicable; and deriving practice-oriented recommendations from direct assessment of trial conduct, external…”
Greaney, C., Cichy, N., O'Sullivan, G., Sills, D., Cahalan, R., & Tierney, A. (2026) · European Respiratory Review · doi:10.1183/16000617.0238-2025 · Cited by 3
“All studies were assessed using the MMAT [27], and this appraisal is evidenced in table 4.”
Jia, H., Kim, D., & Kwon, H. (2026) · Administrative Sciences · doi:10.3390/admsci16050229 · Cited by 3
“Conceptual and system development studies assessed using AACODS showed moderate quality levels (M = 71.8%), while the single mixed-methods study evaluated with MMAT reached the minimum inclusion threshold (60%).”
InfluentialSachdeva, M., Singla, A., & Tanwar, N. (2026) · International Journal of Ethics and Systems · doi:10.1108/ijoes-04-2025-0163 · Cited by 4
Stuckey, H. L., Cooke, D. D., Knehans, A. C., & Skinner, T. C. (2026) · Diabetic Medicine · doi:10.1111/dme.70369 · Cited by 2
“26 Due to the diversity of study methods used in this review (both qualitative and quantitative), the MMAT provided specific criteria for evaluating each method, overcoming the challenge of using separate tools.”
Marcos-Rivero, B., Ortuondo, J., Haegele, J. A., Reina, R., Ascondo, J., Becerril-Atxikallende, I., & Yanci, J. (2026) · European Journal of Special Needs Education · doi:10.1080/08856257.2026.2680128 · Cited by 2
Khanjani, M., Kim, S., Yazdani, H. R., & Sadeghi, M. R. (2026) · European journal of training and development · doi:10.1108/ejtd-01-2026-0010 · Cited by 5
Omoge, A. O., & Omoge, M. O. (2026) · Mental Wellness · doi:10.4081/mw.2026.47 · Cited by 2
Nguyen, H., Chen, H., & Asfani, K. (2026) · Journal of Educational Computing Research · doi:10.1177/07356331261450647 · Cited by 10
The integration of generative artificial intelligence in secondary education: A systematic review
Pérez-Montesdeoca, H., Rodríguez-Rodríguez, D., & Fernández‐Sogorb, A. (2026) · Contemporary Educational Technology · doi:10.30935/cedtech/18746 · Cited by 5
Xiao, N., Li, H., Chen, L., Liu, S., Xiang, Q., & Wang, P. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1840884 · Cited by 3
“A formal critical appraisal of the included studies was conducted using the Mixed Methods Appraisal Tool (MMAT) 2018 (Hong et al., 2018).”
Kumar, V., Vyas, N., Godinho, M. A., Raj, S. E. A., & Govindan, S. (2026) · Discover Public Health · doi:10.1186/s12982-026-02352-9 · Cited by 3
“Qualitative research was appraised with the Critical Appraisal Skills Programme (CASP) Qualitative Checklist, [20] and mixed methods studies were assessed using the Mixed Methods Appraisal Tool (MMAT) [21] .”
Zhai, J., Millar, R., Lam, M., Pont, A., Kelly, R., & de Courten, B. V. (2026) · Australian Journal on Ageing · doi:10.1111/ajag.70200 · Cited by 4
Zarea, H., Shirkoohi, S. M., Ertz, M., & Hessas, D. (2026) · Economies · doi:10.3390/economies14070255 · Cited by 2
“Full-text eligibility assessment used the Mixed Methods Appraisal Tool (Hong et al., 2018) as a standardised quality yardstick across qualitative, quantitative, and mixed-methods studies.”
Value-based teaching practices in cultural contexts: a systematic review
Almajdoub, Z., Khurma, O. A., Alramamneh, Y., et al. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2697519 · Cited by 2
Su, F., Beh, W. F., & Lee, K. C. S. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.107353 · Cited by 9
Alam, N. B., Haque, M. T., Subedar, S., Giacco, D., Singh, S. R. G., & Jilka, S. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1772744 · Cited by 9
“NA, SS and TH assessed the quality of the studies using the Mixed Methods Appraisal Tool (MMAT) (38) as we included quantitative (categorised in the MMAT as ‘quantitative descriptive’, ‘quantitative randomised controlled trials’ and ‘quantitative non-randomised controlled trials’), and mixed methods studies.”
Zhao, Y., Guo, R., Miao, Y., Luo, Y., Wang, H., & Wu, Y. X. (2026) · Journal of Medical Internet Research · doi:10.2196/89190 · Cited by 2
“…quality was appraised for empirical human-participant or evaluator-based studies using the 2018 version of the Mixed Methods Appraisal Tool (MMAT) [30], which provides design-specific criteria for randomized, nonrandomized, quantitative descriptive, qualitative, and mixed methods studies.”
Adult-Child Caregivers’ Motivations when Caring for Home-Dwelling Parents with Dementia
Dombestein, H. (2021) · University of Stavanger eBooks · doi:10.31265/usps.87 · Cited by 3
Allaire, J.-F., Thiam, Y., Morin, P., Zomahoun, H. T. V., Rhéault, N., Lacasse, F., Doré, C., Hyppolite, S.-R., & Garon, S. (2021) · International Journal of Integrated Care · doi:10.5334/ijic.5629 · Cited by 5
Cash Transfers and Child Nutrition: Evidence from sub-Saharan Africa
Awojobi, O. N. (2021) · SocioEconomic Challenges · doi:10.21272/sec.5(3).61-80.2021 · Cited by 3
Patients’ experiences of cognitive impairment following critical illness: a scoping review protocol
Alrø, A. B., Nedergaard, H. K., Svenningsen, H., Jensen, H. I., & Dreyer, P. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-056156 · Cited by 3
Okah, E. D., Glover, L. M., Donahue, K. E., Corbie-Smith, G. M., & Dave, G. (2022) · Journal of General Internal Medicine · doi:10.1007/s11606-022-07737-5 · Cited by 2
“Quality assessment was determined independently by the two lead authors using the Mixed Methods Appraisal Tool (MMAT), version 18.(13) MMAT (see Table 2) evaluates the quality of multiple study designs and has been used in previous mixed-methods systematic reviews.”
MethodAdombire, S., Puts, M. T. E., Ritchie, L. M. P., et al. (2022) · JBI Evidence Synthesis · doi:10.11124/jbies-21-00313 · Cited by 2
A Systematic Review of Children’s Perspectives of Fathers who Perpetrate Intimate Partner Violence
Rogers, K., & Berger, E. (2022) · Trauma Violence & Abuse · doi:10.1177/15248380221124268 · Cited by 6
“Overall, the MMAT indicated the studies included in this review were of good quality (Hong et al., 2018).”
MethodBarro, A., Ngangue, P., Bationo, N., Soubeiga, D., Pafadnam, Y., Kaboré, S. S., Pilabré, A. H., & Traoré, D. (2022) · International Journal of Healthcare · doi:10.5430/ijh.v8n2p39 · Cited by 3
Kelbore, A. G., Enbiale, W., Mosam, A., & Wyk, J. V. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02133-w · Cited by 2
“Selection of MMAT questions: specific criteria to determine the appropriateness for inclusion of each study.”
MethodInfluentialImplementing essential diagnostics-learning from essential medicines: A scoping review
Nyanchoka, M., Mulaku, M., Nyagol, B., Owino, E. J., Kariuki, S., & Ochodo, E. (2022) · PLOS Global Public Health · doi:10.1371/journal.pgph.0000827 · Cited by 5
“Unclear 7 Region North Africa Egypt (1) West Africa Benin, Burkina Faso, Ghana, Mali, Nigeria, Sierra Leone (32) Central Africa Cameroon, Gabon (5) Eastern Africa Ethiopia, Kenya, Madagascar, Rwanda, Sudan, Tanzania, Uganda (85) Southern Africa Botswana, Eswatini, Malawi, Mozambique, Namibia, South Africa, Zambia, Zim…”
BackgroundThe acceptability and utility of Indigenous youth health assessments: a narrative systematic review
Fraser, J., Askew, D. A., Mahoney, R., & Spurling, G. (2023) · Australian Journal of Primary Health · doi:10.1071/py22128 · Cited by 3
Epton, L., Patman, S. M., Coventry, T., Bulsara, C., & Ward, K. D. (2023) · The Journal of Smoking Cessation · doi:10.1155/2023/6544215 · Cited by 4
“Qualitative studies were appraised using the Joanna Briggs Institute (JBI) checklist for qualitative research [16], mixed method studies were appraised using the mixed method assessment tool (MMAT) [17], and descriptive quantitative studies were appraised using the survey appraisal tool from the Center for Evidence-Ba…”
MethodVickers, N., Matthews, A., & Paul, G. (2023) · HRB Open Research · doi:10.12688/hrbopenres.13718.1 · Cited by 4
“Additionally, conducting a MMSR will enable the enhancement, depth and richness of comprehension to facilitate an inclusive understanding of the evidence (Hong et al. , 2018).”
BackgroundAggarwal, M., Hutchison, B. G., Kokorelias, K. M., Mehta, K., Greenberg, L. S., Moran, K. A., Barber, D., & Samson, K. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072076 · Cited by 4
Allana, S., Norris, C. M., Hussain, A., & Clark, A. M. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15756 · Cited by 2
Redero, D., Lázaro, E., Vázquez, N., et al. (2023) · Applied Neuropsychology Adult · doi:10.1080/23279095.2023.2248642 · Cited by 2
Phillips, H., Renda, G., & McDonald, R. L. (2023) · International Journal of Design Creativity and Innovation · doi:10.1080/21650349.2023.2253462 · Cited by 2
“The MMAT discourages the use of summative scores, stating that ‘a single number does not provide information on what aspects of studies are problematic’ (Hong et al., 2018).”
BackgroundMethodInfluentialKei, N., Hassiotis, A., & Royston, R. (2023) · Clinical Child Psychology and Psychiatry · doi:10.1177/13591045231203097 · Cited by 3
“…quality of studies included in the review was assessed using the Mixed Methods Appraisal Tool (MMAT) in 5 aspects: performance of randomisation, group differences at baseline, completeness of outcome data, blinding of outcome assessors and participants ’ adherence rates (Hong et al., 2018).”
MethodSlack, E., Pears, K. A., Rankin, J. M., Newton, J. L., & Pearce, M. S. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-070366 · Cited by 4
“19 All studies were included regardless of quality.”
BackgroundMethodNair, J., Mishra, A., & Sharan, A. M. (2023) · Indian Journal of Palliative Care · doi:10.25259/ijpc_306_2022 · Cited by 2
“[13] The MMAT summarising the various studies included in the present SLR is shown in [Table 1].”
MethodThe impact of music listening intervention on Asia elderly with dementia: a systematic review
Chen, L., Xuanxu, H., Yuyang, X., Zhongqiu, X., & Gao, C. (2023) · European journal of medical research · doi:10.1186/s40001-023-01355-5 · Cited by 5
“In this review, the Mixed Methods Appraisal Tool (MMAT) version 2018 [11] was employed to evaluate the quality of the selected articles.”
MethodTeaching Quantum Physics at Secondary Schools: A Systematic Literature Review
Haron, S., & Halim, L. (2023) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v12-i3/19308 · Cited by 2
“The Mixed-Method Appraisal Tool (MMAT) developed by Hong et al. (2018) was employed for this purpose.”
MethodScience Teaching and Learning Through Digital Education: A Systematic Literature Review
Nor, N. M., & Halim, L. (2023) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v12-i3/19307 · Cited by 2
“The quality assessment phase for the selected studies involved utilizing the Mixed Methods Assessment Tool (MMAT) developed by Hong et al. (2018).”
MethodDevagiri, B., Ariyasena, A. D. K., Siriwardhana, D. D., & Sumanasena, S. (2023) · Child Care Health and Development · doi:10.1111/cch.13192 · Cited by 6
Educational Interventions in Immigrants for Asthma and COPD: A Systematic Review
Al-Juhaishi, M., Lim, C. X., Chan, V. W. S., Stupans, I., & Thrimawithana, T. R. (2023) · Patient Preference and Adherence · doi:10.2147/ppa.s431224 · Cited by 2
Rico Saktiawan Jang Jaya, Abu Hanifah Ayob, Zafir Khan Mohamed Makhbul, & Hussain, W. (2023) · Jurnal Pengurusan · doi:10.17576/pengurusan-2023-68-10 · Cited by 3
Badacho, A. S., & Mahomed, O. H. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294979 · Cited by 2
Cho, H., You, S. B., Hodgson, N. A., Massimo, L. M., & Demiris, G. (2023) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2023.0514 · Cited by 4
“17 This quality assessment tool helps appraise the quality of empirical studies.”
Feasibility in embedding charismatic leadership strategies into social work applications
Grinev, L. (2023) · International Journal of Public Leadership · doi:10.1108/ijpl-07-2023-0057 · Cited by 2
Cannon, P. C., Setia, S. A., Klein-Gardner, S. S., Kavoussi, N., Webster, R. J., III, & Herrell, S. D. (2024) · Journal of Endourology · doi:10.1089/end.2023.0059 · Cited by 3
“46 The MMAT is an established assessment tool that enables the methodological evaluation across five different study categories (randomized controlled trials, nonrandomized controlled trials, quantitative descriptive studies, qualitative studies, and mixed methods studies), making it suitable for…”
Quinn, C., Young, H., Gridley, K., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297747 · Cited by 2
Prioritising patients for semi‐urgent surgery: A scoping review
Coffey, E. K., Walker, R., Nicholson, P., & Gillespie, B. M. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17056 · Cited by 5
Filausová, D., Belešová, R., Trešlová, M., & Červený, M. (2024) · Pielegniarstwo XXI wieku / Nursing in the 21st Century · doi:10.2478/pielxxiw-2024-0006 · Cited by 2
Bab, N. B., Rahman, R. N. A. R. A., Mohamed, S., Radzi, N. A. M., & Yusof, N. (2024) · Special Care in Dentistry · doi:10.1111/scd.12990 · Cited by 6
Rambaldini‐Gooding, D., Keevers, L., Clay, N., & MacLeod, L. (2024) · Australian Journal of Social Issues · doi:10.1002/ajs4.332 · Cited by 4
Optometry student clinical practice at public health facilities: A systematic review
Segooa, R. F., & Moodley, V. R. (2024) · Health SA Gesondheid · doi:10.4102/hsag.v29i0.2441 · Cited by 3
“Mixed Methods Appraisal Tool (MMAT) sourced from Hong et al. (2018a:285–291) and Hong et al. (2018b:1–11) was used to appraise one study, and two other studies were appraised according to the Non-Research Literature Appraisal Tool (Yale New Haven Health 2016:1–3).”
MethodStrategies for Educators to Teach Mixed Methods Research: A Discussion
Younas, A., Durante, Á., Fàbregues, S., & Escalante‐Barrios, E. L. (2024) · The Qualitative Report · doi:10.46743/2160-3715/2024.6464 · Cited by 5
Self-citationMachová, A., Mágrová, M., Hanzl, M., et al. (2024) · Pielegniarstwo XXI wieku / Nursing in the 21st Century · doi:10.2478/pielxxiw-2024-0011 · Cited by 2
Testicular Cancer and Paraneoplastic Encephalitis: A Review of the Current Literature
Deol, E. S., Paracha, A., Paracha, M., Pasha, O. M., Mustafic, E., Davis, R., Siddiqui, Z., Abid, A., Durrani, U. S., Rodriguez, W., et al. (2024) · Clinical Genitourinary Cancer · doi:10.1016/j.clgc.2024.102111 · Cited by 2
Walsh, D. J., O’Driscoll, M., Horgan, A. M., Tabb, E., Hannan, M., Morris, C., & Sahm, L. J. (2024) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2024.04.018 · Cited by 2
Quiñones-Rozo, L. D. P., & Canaval, G. E. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-10780-z · Cited by 4
“The recommendations given in the Mixed Methods Appraisal Tool (MMAT) checklist, version 2018 [13] are followed.”
MethodFactors associated with mosquito control among construction workers: A systematic review
Dapari, R., Fadzil, M. F. M., Hanzir, M. Y., Jais, J. S. M., Safarudin, N. F., & Albar, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303330 · Cited by 3
“The quality appraisal of all four quantitative non-randomised studies in this systematic review was conducted using the MMAT based on five main criteria [10].”
MethodHealth equity in urban and rural settings: implementation of the place standard tool in Spain
Ortiz, A. O., Gea-Caballero, V., Juárez‐Vela, R., Peiró, R., Pérez‐Sanz, E., Santolalla‐Arnedo, I., Sufrate‐Sorzano, T., Garrote‐Cámara, M. E., & Paredes-Carbonell, J. J. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1292032 · Cited by 4
Kelly, Y., Gannon, J., Bassul, C., Williams, M. C., Morrissey, D., McKee, J., Keyes, L. M., O’Rourke, N., & Flynn, R. (2024) · Health & Social Care in the Community · doi:10.1155/2024/6632018 · Cited by 2
“Te Mixed Methods Appraisal Tool (MMAT) was used to assess the quality of empirical studies [33].”
MethodTowards sustainable lean success: a systematic literature review on critical failure factors
Aripin, N. M., Nawanir, G., Hussain, S., & Moshood, T. D. (2024) · Technological Sustainability · doi:10.1108/techs-11-2023-0042 · Cited by 2
Factors related to immigrants’ disaster preparedness: A scoping review
Piao, M., Kondo, A., & Huilin, Q. I. A. N. (2024) · Health Emergency and Disaster Nursing · doi:10.24298/hedn.2023-0008 · Cited by 3
Diagnostic overshadowing of mental health conditions in UK health care
Hale, Z. J. (2024) · Mental Health and Social Inclusion · doi:10.1108/mhsi-04-2024-0060 · Cited by 2
Emergency nurses’ perspectives on mental health boarders: An integrative review
Keslar, R. (2024) · International Emergency Nursing · doi:10.1016/j.ienj.2024.101463 · Cited by 2
Considering occupational performance during recovery of distal radius fracture: A scoping review
Bredy, T. M., Glasgow, C., Liddle, J., Colwell, S., Holding, J., Swan, S., & Patterson, F. (2024) · Australian Occupational Therapy Journal · doi:10.1111/1440-1630.12965 · Cited by 2
Gonzales‐Walters, F., Weldon, S. M., & Essex, R. (2024) · International Journal of Healthcare Simulation · doi:10.54531/rafh4191 · Cited by 9
Simpson, E. L., Sutton, A., Cantrell, A. J., Clowes, M., Weich, S., Bentley-Hollins, K., Visintin, C., & Axtell, C. M. (2024) · Discover Psychology · doi:10.1007/s44202-024-00184-9 · Cited by 6
“Quality of studies was assessed [34, 35] to categorise level of quality as high, medium or low.”
Stokes, K., Piaggio, D., Micco, F. D., Zarro, M., Benedictis, A. D., Tambone, V., Moon, M., Maccaro, A., & Pecchia, L. (2024) · Infectious Disease Reports · doi:10.3390/idr16030039 · Cited by 5
“As regards quality appraisal, since the included studies were of mixed type, namely both qualitative and quantitative, it was decided to recur to the use of the MMAT tool [17], which contains di ff erent subsections that allow for the assessment of quality of di ff erent kinds of studies.”
MethodPaediatric out-of-hospital clinical deterioration: a mixed-methods scoping review protocol
Alghaith, A., Whitley, G. A., Alsuwais, S., Coats, T., & Roland, D. (2024) · BMJ Paediatrics Open · doi:10.1136/bmjpo-2024-002672 · Cited by 2
“28 The purpose of undertaking this quality assessment is to understand and present the methodological rigour and reliability of the current literature.”
Ling, M. Y. J., Ahmad, N., Aizuddin, A. N., & Jaáfar, M. H. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308152 · Cited by 2
Denend, T. V., Moein, S., Niec, P., Peterson, E. W., Morris, J., Backus, D. A., Sosnoff, J. J., Brunskill, A., Hawari, L., & Rice, L. A. (2024) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2024.2394814 · Cited by 2
“…was employed, which is an appropriate tool for appraising systematic mixed studies reviews [21]. this MMAt systematically evaluates the methodological quality of mixed methods studies examining their qualitative and quantitative sections separately, as well as evaluating their integration [21].”
The Principles of Home Care for Patients with Stroke: An Integrative Review
Bahadoram, S., Arsalani, N., Fallahi‐Khoshknab, M., Shahboulaghi, F. M., & Dalvandi, A. (2024) · Iranian Journal of Nursing and Midwifery Research · doi:10.4103/ijnmr.ijnmr_42_23 · Cited by 2
“[28] Disagreements between the first and second authors were resolved through group discussion with the third author.”
BackgroundIbrahim, S., BIELECKI, J. M., Kocabas, E., Singh, S., Senff, J. R., Casaubon, L. K., Rosand, J. M., Rac, V. E., & Pikula, A. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-088631 · Cited by 6
“78 The following describes the information that will be extracted: Quantitative data : (1) general article informa-tion—author’s last name(s), publication date, study setting, country and continent of publication as well as funding agency; (2) study purpose, research design and target population…”
Biomarkers for Early Detection of Stroke: A Systematic Review
Anwar, L., Ahmad, E., Imtiaz, M., Ahmad, B., Ali, M. A., & Mahnoor, M. (2024) · Cureus · doi:10.7759/cureus.70624 · Cited by 10
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT), which is designed to appraise the methodological quality of various study designs [36].”
MethodRamsden, R., Reeves, E., Whitwell, E., Lane, N., & Fletcher, H. K. (2024) · Journal of Intellectual Disabilities · doi:10.1177/17446295241289734 · Cited by 8
“This tool consists of two optional screening questions (included in the current review), to determine if the study is empirical research, and a total of 25 appraisal questions (Hong et al., 2018).”
InfluentialInitial crisis leadership during COVID-19 higher education: a systematic review
Crawford, J., Butler-Henderson, K. A., Lalani, K. H., Rudolph, J., & Sabu, K. M. (2024) · Technology Pedagogy and Education · doi:10.1080/1475939x.2024.2410276 · Cited by 6
“This includes rigorous use of the PRISMA approach, double screening, double full-text screening and single quality assessments (Butler-Henderson et al., 2020; Hong et al., 2018).”
Gjestvang, C., Olsen, M. L., Dalhaug, E. M., & Haakstad, L. A. H. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-086682 · Cited by 3
Kelly, S. B., Stephens, M., Clark, A., Chesterton, L., & Hubbard, L. (2024) · Educational Gerontology · doi:10.1080/03601277.2024.2407201 · Cited by 2
“All included papers were independently assessed for methodological quality by 2 reviewers, using appropriate, validated tools: (1) the Critical Appraisal Skills Programme (2019) and (2) the Mixed Methods Appraisal Tool (Hong et al., 2018).”
MethodMaffissoni, A. L., Martini, J. G., Lazzari, D. D., Vendruscolo, C., Sanes, M. D. S., & Bresolin, P. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7374.4394 · Cited by 2
Steiner, S., Dürk, M., & Warth, M. (2024) · The Arts in Psychotherapy · doi:10.1016/j.aip.2024.102228 · Cited by 2
Applicability and benefits of Standardised Nursing Terminology in Australia: A scoping review
Jedwab, R. M., Holzhauser, K., Raghunathan, K., Lord, Z. K. M., Duncan, S. P., Murray, M. A., Gogler, J., & Hovenga, E. J. S. (2024) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2024.10.001 · Cited by 4
Prescelia, C., Ireland, P. J., Blanco, C., Nguyen, D., & Pacey, V. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2024.2419955 · Cited by 2
“Two reviewers independently completed quality assessment of the selected studies using the Mixed Methods Appraisal Tool (MMAT) – Version 2018 [15].”
Social Support for Adult Community Corrections Offenders: A Systematic Review
Wang, X., Ah, S. H. A. B., & Azman, Z. B. (2024) · Journal of Forensic Psychology Research and Practice · doi:10.1080/24732850.2024.2427798 · Cited by 3
Nieberler-Walker, K., Desha, C., Roiko, A. H., Caldera, S., & Bosman, C. (2024) · Sustainability · doi:10.3390/su162310288 · Cited by 2
Employee adaptation in the return-to-office era within financial institutions
Magliozzi, P. (2024) · SAM Advanced Management Journal · doi:10.1108/samamj-05-2024-0016 · Cited by 3
Turra, L., Lazzari, D. D., Martini, J. G., et al. (2024) · Revista gaúcha de enfermagem · doi:10.1590/1983-1447.2024.20230280.pt · Cited by 2
Uhm, J., & Kim, S. H. (2024) · Nurse Education Today · doi:10.1016/j.nedt.2024.106554 · Cited by 4
Established and emerging roles for pharmacy in operating theatres: a scoping review
McGowan, A., Deasy, E., Coyle, M., et al. (2024) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-024-01845-4 · Cited by 2
“The quality of the evidence was assessed by two reviewers using the Mixed Methods Appraisal Tool (MMAT) [33].”
Hardin, K., Rossi-Katz, J. A., & Busch, S. (2025) · American Journal of Speech-Language Pathology · doi:10.1044/2024_ajslp-24-00126 · Cited by 2
Community Health Workers in Nepal: A Systematic Review of Their Role in Health System Strengthening
Ghimire, S., & Bhatt, P. R. (2025) · Asian Journal of Population Sciences · doi:10.3126/ajps.v4i1.73906 · Cited by 2
“We utilized the Mixed Methods Appraisal Tool (MMAT), version 2018 to assess the methodological quality of the selected studies, as it is designed explicitly for qualitative, quantitative, and mixed methods (Hong et al., 2018).”
Bursik, J., Thiel, A., Mayer, J., & John, J. M. (2025) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2025.2450791 · Cited by 6
“We utilized the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) to evaluate the methodological quality of studies included in our review.”
Brady, G., Clark, S. E., Wagstaff, D., et al. (2025) · Perioperative Care and Operating Room Management · doi:10.1016/j.pcorm.2025.100477 · Cited by 2
Competencies required for the performance of primary health care managers: a systematic review
Soto-Schulz, K., Echenique, R. H., Brito-Díaz, R., & Pérez-Romero, N. (2025) · Cadernos de Saúde Pública · doi:10.1590/0102-311xen092624 · Cited by 11
“Results of methodological quality assessment with the Mixed Methods Appraisal Tool (MMAT).”
InfluentialExploring how the uptake of childhood immunisations can be improved in England: A systematic review
Ames, S., & Kelly, G. (2024) · Practice Nursing · doi:10.12968/pnur.2024.35.12.412 · Cited by 3
Gamage, M. W. K., Pu, L., Moyle, W., Barton, M., & Todorovic, M. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106606 · Cited by 5
Tkaczyk, S.-L. B., Miller, S., Angus, D. J., Lawless, A., Paulson, G., & Watt, B. D. (2025) · Journal of Cross-Cultural Psychology · doi:10.1177/00220221251314431 · Cited by 3
“The Mixed Methods Appraisal Tool (MMAT) Version 18 (Hong et al., 2018) was selected to appraise the quality of included studies within this review.”
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.