The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,401–2,500 of 4,578
Raheel, H., Ferguson, A., Leslie, S., Guardado-Menjivar, V., Chen, K., Merceron, A., Arciniegas, J., Lovvorn, A. E., Higgins, M. K., Barr, D. B., et al. (2026) · Environmental Research Letters · doi:10.1088/1748-9326/ae49a3 · Cited by 7
Implementation of digital governance to enhance oversight in local municipalities
Zvaita, G. T., & Shava, E. (2026) · Frontiers in Sustainable Cities · doi:10.3389/frsc.2026.1713754 · Cited by 4
Liu, X., Li, J., & Kim, S. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1772661 · Cited by 4
“Methodological quality was appraised using the Mixed Methods Appraisal Tool (MMAT, 2018) (28).”
Bartoníčková, D., Chiappinotto, S., Kohanová, D., Gurková, E., Palese, A., Özsaban, A., Bayram, A., Açıl, S. B., Demi̇r, Ö. İ., & Zuzana, S. (2026) · Nurse Education in Practice · doi:10.1016/j.nepr.2026.104783 · Cited by 4
From the Ocular Surface to Neurophysiology: An Integrative Review of Digital Eye Strain
Manna, P. (2026) · Clinical Optometry · doi:10.2147/opto.s586750 · Cited by 7
“24 Strict adherence to the MMAT operational manual minimized subjective bias and ensured consistency during the quality appraisal conducted by a single reviewer.”
Munyangane, O. L., Matshidze, P. E., Lekgothoane, K., Mokwena, T. J., & Negota, V. (2026) · International Journal of Research in Business and Social Science (2147-4478) · doi:10.20525/ijrbs.v15i1.4422 · Cited by 3
“Additionally, each study was appraised for methodological quality using the Mixed Methods Appraisal Tool (MMAT), version 2018 (Hong et al., 2018)”
Meldrum, P., Johnson, B. P., Lo, B. C. Y., Bedelis, M. L., & Rabba, A. S. (2026) · Autism in Adulthood · doi:10.1177/25739581261427260 · Cited by 6
Anantharaman, D., Meyer, C., Nisar, M., Kumaran, S. E., Keay, L. J., McAvoy, S., & Dawes, P. (2026) · The Gerontologist · doi:10.1093/geront/gnag020 · Cited by 4
“( Hong et al., 2018) Based on the study type, the reviewers selected the appropriate criteria category (qualitative, quantitative descriptive, or mixed methods) within the MMAT for each study; (ii) Levels of Evidence (March 2009) was used to assess the strength and reliability of research findings.”
Reclaiming the critical review method: a reconceptualization
Al-Naabi, I., Crawford, J., To, L., & Kelder, J. (2026) · Quality & Quantity · doi:10.1007/s11135-026-02694-1 · Cited by 6
“Among the well-established tools are the Quality assessment with diverse studies (QuADS) (Harrison et al. 2021), and The mixed methods Appraisal Tool (MMAT) (Hong et al. 2018), and Quality assessment tool for theory-based and literature review studies (QATTL) (Crawford et al. 2025), which have published evidence on th…”
Myagmarkhorloo, C., & Habók, A. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2644673 · Cited by 7
Carter, A., Gracey, F., Moody, J., Ovens, A., & Chatburn, E. (2026) · Journal of Social Media Research · doi:10.29329/jsomer.84 · Cited by 7
Global reporting and underreporting of occupational diseases: A systematic review
Khoe, L. C., Saldi, S. R. F., Isbayuputra, M., Mansyur, M., Wiseman, V. L., & Asante, A. D. (2026) · PLoS ONE · doi:10.1371/journal.pone.0345318 · Cited by 13
“At least two reviewers (LK and SR or LK and MI) assessed and appraised the methodological quality of the studies independently using Mixed Methods Appraisal Tool (MMAT) [26].”
Artificial intelligence in doctoral training: Bridging gaps, easing burdens
Rodríguez, G. A., Chiappe, A., & Durand, J. C. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.102744 · Cited by 5
Gidisu, P., Abroampa, W. K., & Nyatsikor, M. K. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.102780 · Cited by 8
Liçenji, L., & Hoxha, J. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1815243 · Cited by 9
Amir-Behghadami, M., & Gholipour, K. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-106406 · Cited by 3
Okorobe, P. M., Onama, J., Bolarin, O. L., Akinola, S. A., Boyetey, S. T., Ampong, J., Kareem, Y. A., Sedor, P. D., Nakachwa, S. M., Ahomugisha, J., et al. (2026) · BMC Public Health · doi:10.1186/s12889-026-28483-x · Cited by 3
“The Mixed Methods Appraisal Tool was used due to heterogeneity of designs of the included studies [28].”
Kopbayeva, M., Uspabayeva, A., & Atıcı, T. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1854373 · Cited by 4
Jiang, X., & Abdullah, Z. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1856440 · Cited by 15
Learning from Experience : The Use of Structured Video-Assisted Debriefing Among Nursing Students
Zhang, H. (2020) · Linköping University medical dissertations · doi:10.3384/diss.diva-172784 · Cited by 3
Best practices in mixed methods for pharmacy and health services research
Rao, D., & Shiyanbola, O. O. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-91888-6.00033-8 · Cited by 3
Nurses’ and Patients’ Perceptions about Psychiatric Intensive Care—An Integrative Literature Review
Berg, J., Gustafsson, N., Mishina, K. E., Ellilä, H., & Lahti, M. (2022) · Issues in Mental Health Nursing · doi:10.1080/01612840.2022.2101079 · Cited by 5
Music in the acute preoperative nursing care — A mixed‐method pilot study
Antonsen, L. K., Dieperink, K. B., & Østervang, C. (2022) · Nursing Open · doi:10.1002/nop2.1516 · Cited by 5
“To appraise the methodological quality, this study followed the Mixed Method Appraisal Tool (Hong et al., 2018).”
MethodMorin, É., Murray, J., Rioux, C., & Ryan, N. C. (2022) · Revue de psychoéducation · doi:10.7202/1093464ar · Cited by 6
“Selon les recommandations d’Houle et al. (2019), la qualité des études a été analysée avec l’outil d’évaluation Mixed Methods Appraisal Tool (MMAT-Version 2018; Hong et al., 2018), qui inclut des énoncés selon différents devis.”
Oceja, J., Villanueva‐Blasco, V. J., Vázquez‐Martínez, A., Villanueva-Silvestre, V., & Al‐Halabí, S. (2023) · Sustainability · doi:10.3390/su15021456 · Cited by 5
Hammond, M., Ryan, C., Garvey, A., & Dwyer, A. (2023) · Journal of EMDR Practice and Research · doi:10.1891/emdr-2022-0020 · Cited by 4
Sandsten, A., Lindgren, B., Strömbäck, M., Ejneborn-Looi, G.-M., & Gabrielsson, S. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076437 · Cited by 3
Impact of Hearing Loss on Social Participation in Children: A Scoping Review
Aggarwal, K., Ravi, R., & Yerraguntla, K. (2023) · Indian Journal of Otolaryngology and Head & Neck Surgery · doi:10.1007/s12070-023-04284-1 · Cited by 8
“The Mixed Methods appraisal tool (MMAT) version 2018 [18] was used to assess the quality of included studies.”
Adult male rape myths in England since 1994: A systemic mixed methods review.
Kambashi, N., Rechdan, J., Noon, E., & Wilson, A. D. (2023) · Psychology Public Policy and Law · doi:10.1037/law0000402 · Cited by 6
Tamborini, M. M. D. F., Centenaro, A. P. F. C., Souto, E. N. D. S., Andres, A. T. G., Stumm, E. M. F., & Colet, C. D. F. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6797.4042 · Cited by 3
Parental Mental Illness in the Family Courts: A Scoping Review
Holford, T. A., Reupert, A., Tchernegovski, P., & Rhoades, H. (2023) · Journal of Child and Family Studies · doi:10.1007/s10826-023-02708-8 · Cited by 7
“The Mixed Methods Appraisal Tool (MMAT) was used to assess the quality of the papers included (Hong et al., 2018).”
MethodImplementation of Hospital Mortality Reviews: A Systematic Review
Schönfeld, M. S., Härter, M. C., Schröder, A. S., Kokartis, K., Bartz, H.-J., & Kriston, L. (2024) · Journal of Patient Safety · doi:10.1097/pts.0000000000001187 · Cited by 3
Elkefi, S., Blecker, S., & Bitan, Y. (2024) · Applied Clinical Informatics · doi:10.1055/s-0043-1776792 · Cited by 5
“32 The MMAT assesses the quality of qualitative, quantitative, and mixed-methods studies.”
BackgroundInfluentialImplementation of Group-Based Exercise Programs in Physiotherapy
Berre, M. L., Forest, L., & Dumoulin, C. L. (2024) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000002477 · Cited by 3
“11 Again, all discrepancies were resolved through consensus or by involving a third reviewer (CD).”
Budiman, I., Wagemakers, A., Derks, M., Sahara, S., & Moore, S. (2024) · NJAS Impact in Agricultural and Life Sciences · doi:10.1080/27685241.2024.2349518 · Cited by 6
“To ensure the 20 articles are trustworthy, valid and reliable, three authors (IB, MD, AW) conducted a quality assessment, using the appraisal tools from Hong, et al (Hong et al., 2018).”
Baker, J. A., Kendal, S., Bojke, C., Louch, G., Halligan, D., Shafiq, S., Sturley, C., Walker, L. E., Brown, M., Berzins, K. M., et al. (2024) · Health and Social Care Delivery Research · doi:10.3310/udbq8402 · Cited by 4
Identifying Learning Preferences and Strategies in Health Data Science Courses: Systematic Review
Rohani, N., Sowa, S., & Manataki, A. (2024) · JMIR Medical Education · doi:10.2196/50667 · Cited by 6
“In addition, both reviewers assessed the quality of the included articles independently by using the MMAT [49].”
Chong, G. Y., Kaur, S., Ruzita, A. T., Hoe, W. M. K., Tan, H. Y., Loy, S. L., Chen, L., Siah, W. Y., Lim, G. P., Chee, Y. Y., et al. (2024) · Chronobiology International · doi:10.1080/07420528.2024.2360742 · Cited by 4
“Domain criteria were grouped as follows: (i) score ≤ 2; (ii) score = 3; (iii) score ≥ 4 (out of 5) represented as low, medium and high risk of bias if they met the respective scores(Nha Hong et al. 2018).”
BackgroundMethodDolatabadi, Z., Moridi, M., Farnam, F., & Damghanian, M. (2024) · Reproductive Health · doi:10.1186/s12978-024-01816-y · Cited by 3
“The methodological quality of the included studies will be assessed using the Mixed Methods Appraisal Tool (MMAT), a reliable and validated tool specifically designed for systematic mixed studies reviews [29].”
MethodFerrone, N. G., Sanmartin, M. X., O’Hara, J., et al. (2024) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2024.07.026 · Cited by 3
Somma, R. D., & Fleming, P. (2024) · Brain Injury · doi:10.1080/02699052.2024.2392161 · Cited by 3
Chowdhury, N. A., Erman, D., Raihan, M. M. H., Marshall, Z., Datta, R. K., Aghajafari, F., Shankar, J., Sehgal, K., Rashid, R., & Turin, T. C. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-077113 · Cited by 5
Molla, A., Usher, K. J., Maharaj, R., & Islam, M. S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310255 · Cited by 5
Beyond error: A qualitative study of human factors in serious adverse events
Mujuru, C., & Peisah, C. (2024) · Journal of Healthcare Risk Management · doi:10.1002/jhrm.21583 · Cited by 3
How is recovery defined and measured in patients with low back pain? A mixed study systematic review
Lukacs, M. J., Peters, N., Minetama, M., Kowalski, K. L., Stanley, M., Jayaprakash, K., Walton, D. M., & Rushton, A. (2024) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-024-07892-2 · Cited by 3
“The MMAT 2018 tool is a quality assessment tool first created in 2006 for multiple study designs including qualitative, randomized control trials (RCTs), quantitative non-randomized, quantitative descriptive and mixed methods studies [17].”
MethodEffect of telemedicine interventions on weight loss: A systematic review and meta-analysis
Moulaei, K., Moulaei, R., Bahaadinbeigy, K., & Bastaminejad, S. (2024) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x241273991 · Cited by 3
“Two researchers independently used the Mixed Methods Appraisal Tool (MMAT, version 2018: Hong et al., 2018) for a thorough data assessment.”
Egli, V., McIntyre, E., & Duggan, N. (2024) · Contemporary Nurse · doi:10.1080/10376178.2024.2409737 · Cited by 3
“Therefore, readers can be con fi dent that our fi ndings are reliable (Hong et al., 2018).”
Chapman, H. M., McSherry, R., Bettany‐Saltikov, J., et al. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1446 · Cited by 3
“All qualitative primary research studies will be assessed using the MMAT tool (Hong et al., 2018) and synthesised using thematic analysis according to Braun and Clarke's method (Braun & Clarke, 2022).”
MethodOptimizing classroom environments for visually impaired school children a scoping review protocol
Najmee, N. A. A., Mohammed, Z., Rahman, M. H. A., Fadzil, N. M., & Ludin, A. F. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308149 · Cited by 6
Smith, L. H., Aguilar, L. N., Joshua, K., Pandey, T., Sox, D. M., Hernandez, B. E., Wang, Y., Yang, K., & Bottiani, J. H. (2024) · Journal of School Psychology · doi:10.1016/j.jsp.2024.101402 · Cited by 10
Turra, L., Lazzari, D. D., Martini, J. G., et al. (2024) · Revista gaúcha de enfermagem · doi:10.1590/1983-1447.2024.20230280.en · Cited by 3
Lee, J. W. K., Mowat, R. M., Blamires, J., & Foster, M. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.16716 · Cited by 5
“The methodological quality of the randomised control trials included in this review was evaluated with the Mixed Methods Appraisal Tool (MMAT) for randomised controlled trials to exclude low-quality studies and identify the strengths and limitations of the included studies (Hong et al. 2018).”
Kavanagh, M., Moore, A., & Fassbender, C. (2025) · International Journal of Psychology · doi:10.1002/ijop.70007 · Cited by 4
Mehrabifar, A., Manias, E., & Nicholson, P. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.16759 · Cited by 5
Advanced clinical practitioners' untapped potential to become managers
Addison, E., & Scott, V.-J. (2025) · British Journal of Nursing · doi:10.12968/bjon.2024.0139 · Cited by 3
Karim, M. Z., Al‐Mamun, M., Eva, M. A., Ali, M. H., & Kalam, A. (2025) · The Family Journal · doi:10.1177/10664807251322169 · Cited by 3
Haseldine, C., Pallin, J. A., Kearney, P. M., McHugh, S., Riordan, F., Cotterill, S., & O’Donoghue, G. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-090105 · Cited by 4
“32”
Hamiduzzaman, M., McLennan, V., Gaffney, H. J., Miles, S., Crook, S., Grove, L., Gray, M., & Flood, V. M. (2025) · Health Policy · doi:10.1016/j.healthpol.2025.105304 · Cited by 7
Gardening initiatives as an approach to cancer prevention for children and youth
Gillies, C., & Baay, C. (2025) · Public Health · doi:10.1016/j.puhe.2025.03.015 · Cited by 4
Xu, T., Chen, C., Liu, C., & Rose, G. M. (2025) · Geriatric Nursing · doi:10.1016/j.gerinurse.2025.03.016 · Cited by 5
McKinlay, A. R., Antonopoulou, V., Schenk, P. M., Lorencatto, F., Oliver, E. J., Vlaev, I., Kelly, M. P., Sniehotta, F. F., & Chater, A. (2025) · Health Psychology Review · doi:10.1080/17437199.2025.2485094 · Cited by 4
“Two reviewers (ARM, VA) independently carried out a quality assessment using the Mixed Methods Appraisal Tool (MMAT) on 100% of all 18 articles included within the review (Hong et al., 2018).”
Kim, H.-J., Choi, B., Choi, E., Kim, H., Jang, S., Lim, J., Bulkeley, K., & Smith‐Merry, J. (2025) · Disability and health journal · doi:10.1016/j.dhjo.2025.101854 · Cited by 3
Ronchi, S., Caruso, R., Fabrizi, D., Luciani, M., Alfes, C. M., Colley, N., Magon, A., Conte, G., Valli, A., Terzoni, S., et al. (2025) · MethodsX · doi:10.1016/j.mex.2025.103391 · Cited by 4
“Mixed-method studies, if included, will be assessed using the Mixed Methods Appraisal Tool (MMAT) [ 38 ].”
The Role of Forgiveness for Aging and Advanced Illness: An Integrative Systematic Review
Rossy, C., Pergolizzi, D., Fernández-Capo, M., Martos, C., Gámiz, M., & Crespo, I. (2025) · Journal of Holistic Nursing · doi:10.1177/08980101251346072 · Cited by 4
Eaton, A. D., Scheadler, T. R., Kon, T., Pang, N., Kwan, S., McDonald, M., Dillon, F. R., McInroy, L. B., Beer, O. W. J., Beckwell, E., et al. (2025) · Sexuality Research and Social Policy · doi:10.1007/s13178-025-01147-y · Cited by 4
“Finally, the intervention study was appraised at 80%.”
InfluentialHobbs, L. J., Hertel, J., Walton, L., Schartau, P., & Bailey, J. (2025) · Sexual & Relationship Therapy · doi:10.1080/14681994.2025.2508826 · Cited by 3
“The Mixed Methods Appraisal Tool (MMAT) was used to assess risk of bias for the remaining mixed method, quantitative, and qualitative studies (Hong et al., 2018).”
Leon, F., Rojas, L., Peña, Á., et al. (2025) · Mathematics · doi:10.3390/math13152456 · Cited by 3
Cooke, K., Ridgway, K., Pecora, L. A., Westrupp, E. M., Hedley, D., Hooley, M., & Stokes, M. A. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251357618 · Cited by 5
“We summarized the study quality of each included study, rather than excluding studies of low methodological quality, which is discouraged (Hong et al., 2018).”
Systems Factors Contributing to Racial/Ethnic Disparities in Maternal Health: A Systematic Review
Akintunde, T. Y., Senior, M. A., Logan, A., & Alfred, M. C. (2025) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-025-02583-7 · Cited by 4
“The Mixed Method Appraisal Tool (MMAT) was used to assess the risk of bias of each study [43].”
Krah, N. M., Fai, C., Sutton, K. N., Rader, A., Barton, K. D., Skalla, L. A., Boucher, H. W., Collyar, D. E., Miller, J., Oakes, M., et al. (2025) · Clinical Infectious Diseases · doi:10.1093/cid/ciaf296 · Cited by 3
Lin, Y., Fan, S., Chai, W., Zheng, N., Yu, D., Yang, L., & Chen, L. (2025) · Supportive Care in Cancer · doi:10.1007/s00520-025-09851-6 · Cited by 7
“(4) Mixed-methods studies: Mixed Methods Appraisal Tool (MMAT) with equal weighting of qualitative/quantitative components (percentage score) [33].”
A Systematic Review of the Relation between Complementary Feeding and Children’s Development
Prete, A. D., Galloway, A. T., Farrow, C. V., Bellagamba, F., & Addessi, E. (2025) · Current Nutrition Reports · doi:10.1007/s13668-025-00692-7 · Cited by 13
“A complete description of the scale can be found in [ 68 ].”
InfluentialSchwalbach, T., Riguzzi, M., Köhler, M., & Naef, R. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.70193 · Cited by 7
“The Mixed-Methods Appraisal Tool (MMAT) Version 2018 was used for quality assessment (Hong et al. 2018).”
Iverson, T., Schulze, M., Gill, K., Switkowski, Y., dos Santos, Í. K., & Ashe, M. C. (2025) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2025.106022 · Cited by 6
Espitia-Cepeda, C., Amado, B. G., Algado, S. S., & Villanueva‐Blasco, V. J. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1661844 · Cited by 10
Capper, T. S., Moran, L., Gupta, M., Seaman, G., Cole, C., Grota, T., O'Reilly, R., Jacob, A. M., & Jacob, E. R. (2025) · Nurse Education in Practice · doi:10.1016/j.nepr.2025.104609 · Cited by 4
Alves, E., Rodrigues, R., da Fonseca, C. J. V., Lopes, M. J., Faria, I., Dalkner, N., Barach, P. R., Folkvord, F., Carlson, J. I., Reininghaus, E. Z., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-097330 · Cited by 3
“39 Once again, this step will be carried out by the same two reviewers independently and any disagreement with the evaluation of the quality of the studies must be resolved once again using the third reviewer.”
Belo, J. D. C., Calheiros, T., & Pereira, M. G. (2026) · Climate · doi:10.3390/cli14070148 · Cited by 3
“…review included heterogeneous evidence sources (peer-reviewed articles, technical reports, policy documents, and institutional publications); a formal risk-of-bias assessment using standardised appraisal tools (e.g., RoB 2, ROBINS-I, or JBI instruments) was not considered appropriate [47,54,60].”
Hemşirelik ve Tıp Öğrencilerinin Klinik Öğretiminde 360° Değerlendirme: Kapsam Odaklı İnceleme
Bayzat, Ç., & Kılıkçıer, Ş. S. (2021) · Hacettepe Üniversitesi Hemşirelik Fakültesi Dergisi · doi:10.31125/hunhemsire.966317 · Cited by 4
Karacsony, S., Martyn, J. K., Rosenberg, J. P., & Andrews, S. (2021) · Progress in Palliative Care · doi:10.1080/09699260.2021.2000807 · Cited by 3
Millerd, E., Fisher, A. R., Lambert, J. M., & Pfaff, K. (2021) · Canadian Journal of Nursing Research · doi:10.1177/08445621211067378 · Cited by 3
“The MMAT criteria and detailed explanations for their application are publicly available.”
BackgroundMethodInfluentialProviders’ Experience of Abortion Care: Protocol for a Scoping Review
Dempsey, B., Callaghan, S. L., & Higgins, M. (2022) · JMIR Research Protocols · doi:10.2196/35481 · Cited by 3
A Systematic Review on Supply Chain Risk Management: Issues, Challenges, and Future Agenda
Ali, M. (2022) · PAKISTAN LANGUAGES AND HUMANITIES REVIEW · doi:10.47205/plhr.2022(6-ii)14 · Cited by 7
“Screened studies were further refined through a quality review tool known as the "Mixed Methods Appraisal Tool (MMAT)" (Hong et al., 2018).”
MethodTorres‐Castaño, A., Delgado, J., del Pino‐Sedeño, T., Malaret, D. E., de Pascual, A. M., Toledo‐Chávarri, A., & Serrano‐Aguilar, P. (2022) · European Eating Disorders Review · doi:10.1002/erv.2909 · Cited by 4
Fields, A. M., Linich, K., Thompson, C. M., Saunders, M., Gonzales, S. K., & Limberg, D. (2022) · Counseling Outcome Research and Evaluation · doi:10.1080/21501378.2022.2069555 · Cited by 6
“To assess the 18 articles, we utilized the Mixed Methods Appraisal Tool (MMAT) to determine the quality of each study (Hong et al., 2018).”
MethodInfluentialGANBAT, D., Jugder, B., Ganbat, L., Tomoeda, M., Dungubat, E., Miegombo, A., Garmaa, G., Takahashi, Y., Fukuzawa, R., Mori, I., et al. (2022) · ONCOLOGIE · doi:10.32604/oncologie.2022.022299 · Cited by 4
Neonatal skin injury scales: a scoping review with narrative synthesis protocol
Hall, S., August, D. L., Coyer, F. M., & Marsh, N. M. (2022) · Wound Practice and Research · doi:10.33235/wpr.30.3.175-178 · Cited by 3
Systematic review: Effect of mobile applications on college students with depressive symptoms
Lin, Y.-H. P., & Lou, M. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15716 · Cited by 4
Educational Needs of Rural Nurses when Entering Practice
Corner, S., Dahlke, S., & Hunter, K. F. (2023) · Online Journal of Rural Nursing and Health Care · doi:10.14574/ojrnhc.v23i1.723 · Cited by 2
Dıkeç, G., & Özer, D. (2023) · Issues in Mental Health Nursing · doi:10.1080/01612840.2023.2212768 · Cited by 3
Needs for Discharge Education of Family Caregivers of Pediatric Patients: An Integrative Review
Lee, H., Lee, S. E., & Lee, S. H. (2023) · Journal of Korean Academy of Fundamentals of Nursing · doi:10.7739/jkafn.2022.30.2.137 · Cited by 3
“The MMAT quality scores ranged from 60% to 100%.”
InfluentialDlamini, K., Moetlhoa, B., Turner, A. C., Maluleke, K., & Mashamba-Thompson, T. P. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281849 · Cited by 3
Batley, A., Sewell, P., & Dyer, B. T. J. (2023) · Prosthetics and Orthotics International · doi:10.1097/pxr.0000000000000241 · Cited by 3
Rodríguez‐Prat, A., Pergolizzi, D., Crespo, I., & Monforte‐Royo, C. (2023) · Behavioral Medicine · doi:10.1080/08964289.2023.2214717 · Cited by 5
“The Mixed Methods Appraisal Tool (MMAT) 23 guideline was used to assess the quality of the mixed-methods studies included in the review.”
Kamarrudin, H., Talib, O., Kamarudin, N., & Ismail, N. (2023) · Malaysian Journal of Social Sciences and Humanities (MJSSH) · doi:10.47405/mjssh.v8i6.2342 · Cited by 6
“Source: Hong et al. (2018) All authors concurred that all chosen publications fulfilled the minimal quality standard in terms of technique and analysis as a result of this procedure.”
BackgroundMethodInfluentialA rapid review of initiatives to address financial strain and wellbeing in high-income contexts
Yashadhana, A., Glenn, N. M., Jaques, K., Belon, A. P., Harris, P. J., de Leeuw, E., & Nykiforuk, C. I. J. (2023) · Public Health Research & Practice · doi:10.17061/phrp3322315 · Cited by 3
Collins, T. L., Cardella, A., & Gordon, S. (2023) · Home Healthcare Now · doi:10.1097/nhh.0000000000001187 · Cited by 5
Voermans, M. A. C., Wilthagen, T., & Embregts, P. J. C. M. (2023) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-023-10128-0 · Cited by 6
“Subsequently, the quality of the eligible studies (n = 27) was assessed using the Mixed-Method Appraisal Tool (MMAT), version 2018 [33], which is applicable for the assessment of different types of research designs.”
MethodFive-factor model of personality and Parkinson’s Disease: a systematic review
Arten, T. L. D. S., & Hamdan, A. C. (2023) · Geriatrics Gerontology and Aging · doi:10.53886/gga.e0230022 · Cited by 4
Ekanayake, S., Ward, A., Heart, D., Valery, P. C., & Soar, J. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-068927 · Cited by 3
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.