The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 601–700 of 4,578
Factors Affecting Human Papillomavirus Vaccination in Men: Systematic Review
Shin, H., Jeon, S., Cho, I., & Park, H. (2022) · JMIR Public Health and Surveillance · doi:10.2196/34070 · Cited by 34
“Study Quality Assessment We used the Mixed Methods Appraisal Tool (MMAT) version 2018 for quality assessment of the eligible articles [29].”
MethodVizer, L. M., Mikles, S. P., & Piepmeier, A. T. (2022) · Psycho-Oncology · doi:10.1002/pon.5980 · Cited by 33
Doyle, R., Craft, P., Turner, M. R., & Paterson, C. C. (2022) · Journal of Cancer Survivorship · doi:10.1007/s11764-022-01219-7 · Cited by 36
“The methodological quality assessment was conducted using the Mixed Methods Assessment Tool (MMAT) [29].”
MethodEaspaig, B. N. G., Reynish, Hoang, Bridgman, H., Corvinus-Jones, & Auckland (2022) · Rural and Remote Health · doi:10.22605/rrh6999 · Cited by 46
Sung, M., He, J., Zhou, Q. X., Chen, Y., Ji, J. S., Chen, H., & Li, Z. (2022) · Journal of Medical Internet Research · doi:10.2196/37338 · Cited by 38
“Joanna Briggs Critical Appraisal Tools were used for qualitative research [30], the Mixed Methods Appraisal Tool [31] was used for mixed methods studies, and the Center for Evidence-Based Management Critical Appraisal Checklist was used for cross-sectional studies [32].”
MethodGereis, J. M., Hetherington, K., Ha, L., Robertson, E. G., Ziegler, D. S., Barlow-Stewart, K. K., Tucker, K., Marron, J. M., & Wakefield, C. E. (2022) · European Journal of Human Genetics · doi:10.1038/s41431-022-01170-2 · Cited by 33
“Article quality assessment We assessed eligible studies for quality and risk of bias using the Mixed Methods Appraisal Tool (MMAT) (Supplementary Table 2) [17].”
MethodSuicide in the Construction Industry: A Targeted Meta-analysis
Tyler, S., Hunkin, H., Pusey, K., Gunn, K. M., Clifford, B., & Procter, N. G. (2022) · Archives of Suicide Research · doi:10.1080/13811118.2022.2131488 · Cited by 28
“Studies were assessed for quality using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodYim, S. H., Lorenz, H., & Salkovskis, P. M. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231156198 · Cited by 39
“The methodological quality of included studies was assessed by SHY using the Mixed-Method Appraisal Tool (MMAT; Hong et al., 2018).”
BackgroundMethodInfluentialWatts, T. E., Sydor, A., Whybrow, D., Temeng, E., Hewitt, R. M., Pattinson, R., Bundy, C., Kyle, R. G., & Jones, B. E. (2023) · Nursing Open · doi:10.1002/nop2.1913 · Cited by 36
“The MMAT was constructed specifically for quality appraisal in mixed studies reviews (Hong et al., 2018; Pace et al., 2012).”
MethodNiedzwiedz, C. L., Kankawale, S. M., & Katikireddi, S. V. (2023) · medRxiv · doi:10.1101/2023.12.19.23300198 · Cited by 28
Chutiyami, M., Cutler, N. A., Sangon, S., Thaweekoon, T., Nintachan, P., Napa, W., Kraithaworn, P., & River, J. (2025) · Journal of Primary Care & Community Health · doi:10.1177/21501319251332723 · Cited by 57
“32 Differences between the 2 reviewers were resolved by discussion between the reviewers.”
Vaping during pregnancy: a systematic review of health outcomes.
Ussher, M., Fleming, J., & Brose, L. (2024) · BMC pregnancy and childbirth · doi:10.1186/s12884-024-06633-6 · Cited by 27
“We assessed study quality using the Mixed Methods Appraisal Tool (MMAT) [30].”
MethodDolan, N., Grealish, A., Tuohy, T., & Bright, A. (2022) · Journal of Midwifery & Women s Health · doi:10.1111/jmwh.13335 · Cited by 35
Reece, S., Sheldon, T., Dickerson, J., & Pickett, K. E. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114746 · Cited by 30
Wylie, G., Kroll, T., Witham, M. D., & Morris, J. H. (2022) · Disability and Rehabilitation · doi:10.1080/09638288.2022.2118869 · Cited by 31
Kuruppu, N. R., Chaboyer, W. P., Abayadeera, A., & Ranse, K. (2023) · Australian Critical Care · doi:10.1016/j.aucc.2022.12.009 · Cited by 35
Kadariya, S., Ball, L., Chua, D., Ryding, H. G., Hobby, J., Marsh, J., Bartrim, K., MITCHELL, L. J., & Parkinson, J. E. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20075341 · Cited by 28
O’Sullivan, R., Bissell, S., Hamilton, A., Bagshaw, A. P., & Richards, C. (2023) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2023.101814 · Cited by 34
Patient experience of emergency department triage: An integrative review
Janerka, C., Leslie, G. D., & Gill, F. J. (2024) · International Emergency Nursing · doi:10.1016/j.ienj.2024.101456 · Cited by 46
Teaching and learning clinical reasoning skill in undergraduate medical students: A scoping review
Delavari, S., Barzkar, F., Rikers, R. M. J. P., Pourahmadi, M., Arabshahi, S. K. S., Keshtkar, A., Dargahi, H., Yaghmaei, M., & Monajemi, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309606 · Cited by 48
Campbell, J., Lavoie, L., Farraia, M., Huelin, R., Zhang, Q., & Monfared, A. A. T. (2024) · Neurology and Therapy · doi:10.1007/s40120-024-00676-9 · Cited by 35
van der Mee, F. A. M., Ottenheijm, R. P. G., Gentry, E. G. S., Nobel, J. M., Zijta, F. M., Cals, J., & Jansen, J. (2024) · European Radiology · doi:10.1007/s00330-024-11165-w · Cited by 44
“The quality assessment tool (MMAT) evaluates studies based on fi ve criteria per study design, with each criterion allocated a score of 20% if present [40].”
Kim, Y. H., & Kim, D. (2024) · BMC Nursing · doi:10.1186/s12912-024-02538-z · Cited by 52
Foucault‐Fruchard, L., Michelet-Barbotin, V., Leichnam, A., Tching-Sin, M., Nizet, P., Tollec, S., Nativel, F., Vene, E., Fronteau, C., & Huon, J. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06338-6 · Cited by 41
“The quality and risk of bias of each included study were assessed using the Mixed Methods Appraisal Tool (MMAT) [22] .”
Chen, H., Chen, Y., Zheng, A., Xin, T., & Han, L. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06780-0 · Cited by 30
A systematic review of early evidence on generative AI for drafting responses to patient messages
Hu, D., Guo, Y., Zhou, Y., Flores, L., & Zheng, K. (2025) · npj Health Systems · doi:10.1038/s44401-025-00032-5 · Cited by 40
Schmitt, M. E., Hawkins, M. M., & Florsheim, P. (2025) · Implementation Science Communications · doi:10.1186/s43058-025-00712-1 · Cited by 41
“The Mixed Methods Appraisal Tool (MMAT) version 2018 was used to assess the methodological quality of the included articles [23].”
Are There Changes in Inequalities in Injuries? A Review of Evidence in the WHO European Region
Sengoelge, M., Leithaus, M., Braubach, M., & Laflamme, L. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16040653 · Cited by 31
“To overcome this, we attempted to use the Mixed Methods Appraisal Tool (MMAT) [70] in which all studies rated positively on the seven criteria, indicating an acceptable quality level.”
MethodBerre, M. L., Presse, N., Morin, M., Larouche, M., Campeau, L., Hu, Y., Reid, I., & Dumoulin, C. L. (2020) · Neurourology and Urodynamics · doi:10.1002/nau.24344 · Cited by 33
The experiences of foreign‐educated nurses in Japan: a systematic review
Abuliezi, R., Kondo, A., & Qian, H. L. (2020) · International Nursing Review · doi:10.1111/inr.12640 · Cited by 27
Feeley, C., Crossland, N., Betrán, A. P., Weeks, A. D., Downe, S. M., & Kingdon, C. (2021) · Reproductive Health · doi:10.1186/s12978-021-01146-3 · Cited by 27
“Quality assessment of mixed methods studies was carried out using the Mixed Methods Appraisal Tool (MMAT) tool [13].”
MethodHeafala, A., Ball, L., Rayner, J., & MITCHELL, L. J. (2021) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.12927 · Cited by 30
“It allows appraisal of methodological quality of qualitative research, quantitative descriptive studies and mixed methods using a rating system out of five key criterion (27) .”
BackgroundMethodInfluentialMaluleke, K., Musekiwa, A., Kgarosi, K., Gregor, E. M., Dlangalala, T., Nkambule, S. J., & Mashamba-Thompson, T. P. (2021) · Diagnostics · doi:10.3390/diagnostics11122299 · Cited by 36
“Using MMAT, we appraised the methodological quality of five categories of research: qualitative research, randomised controlled trials, nonrandomised studies, quantitative descriptive studies and mixed methods studies [20].”
BackgroundMethodA rapid review of end-of-life needs in the LGBTQ+ community and recommendations for clinicians
Lintott, L., Beringer, R., Do, A., & Daudt, H. (2022) · Palliative Medicine · doi:10.1177/02692163221078475 · Cited by 35
Yoong, S. Q., Liao, A. W. X., Goh, S. H., & Zhang, H. (2022) · Nurse Education Today · doi:10.1016/j.nedt.2022.105376 · Cited by 37
Piaggio, D., Zarro, M., Pagliara, S. M., Andellini, M., Almuhini, A., Maccaro, A., & Pecchia, L. (2023) · American Journal of Infection Control · doi:10.1016/j.ajic.2023.03.005 · Cited by 36
“The quality appraisal was conducted using the MMAT tool [14], due to the mixed types of studies considered.”
MethodSutherland, D., Flynn, S., Kurzeja, O., Griffin, J., & Hastings, R. P. (2023) · Journal of Intellectual Disability Research · doi:10.1111/jir.13068 · Cited by 40
“Given that a wide range of study designs were eligible for inclusion, we assessed methodological quality using the Mixed Methods Appraisal Tool (MMAT; Hong et al .”
MethodHealth service improvement using positive patient feedback: Systematic scoping review
Lloyd, R., Munro, J., Evans, K., Gaskin-Williams, A., Hui, A., Pearson, M., Slade, M., Kotera, Y., Day, G., Loughlin-Ridley, J., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0275045 · Cited by 43
“The quality of included documents was assessed using the Mixed Methods Appraisal Tool (MMAT) [41] and scores were included in the DAT.”
MethodJoo, Y., Jang, Y., & Kwon, O. Y. (2024) · Nursing in Critical Care · doi:10.1111/nicc.13105 · Cited by 35
Kennedy, C., & McLoughlin, A. (2022) · Early Childhood Education Journal · doi:10.1007/s10643-021-01291-1 · Cited by 60
The “Face” of Coach Development: A Systematic Review of the Role of the Coach Developer
Jones, T., Allen, J. B., & MacDonald, S. C. (2023) · International Sport Coaching Journal · doi:10.1123/iscj.2022-0017 · Cited by 35
Malviya, S. (2023) · Journal of Religion and Health · doi:10.1007/s10943-023-01761-3 · Cited by 33
Room of horrors simulation in healthcare education: A systematic review
Lee, S. E., Repsha, C. L., Seo, W. J., Lee, S. H., & Dahinten, V. S. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105824 · Cited by 37
Bannatyne, A. J., Jones, C., Craig, B. M., Jones, D., & Forrest, K. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1204274 · Cited by 41
“The MMAT allows for evaluation of qualitative studies, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed-method studies.”
MethodInfluentialGoodwin, J., Behan, L., & O’Brien, N. (2021) · Journal of Child and Adolescent Mental Health · doi:10.2989/17280583.2023.2229876 · Cited by 40
Opoku, R., Ackon, S. K., Kumah, E., Botchwey, C. O., Appiah, N. E., Korsah, S., & Peprah, M. A. (2023) · BMC Endocrine Disorders · doi:10.1186/s12902-023-01508-x · Cited by 38
“The tool can appraise five different categories of study designs: qualitative, randomized controlled trial, non-randomized, quantitative descriptive and mixed methods studies [19].”
MethodPediatric produce prescription initiatives in the U.S.: a scoping review
Muleta, H., Fischer, L. K., Chang, M., Kim, N., Leung, C. W., Obudulu, C., & Essel, K. D. (2023) · Pediatric Research · doi:10.1038/s41390-023-02920-8 · Cited by 28
“A quality assessment of included studies was conducted using the Mixed Method Appraisal Tool (MMAT), 38 which uses fi ve criteria related to the appropriateness of the study design, representativeness of the population, adherence to the stated intervention, completeness of outcome data, and…”
MethodHydinger, K. R., Wu, X., Captari, L. E., & Sandage, S. J. (2024) · Pastoral Psychology · doi:10.1007/s11089-024-01150-x · Cited by 46
“We utilized the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) to evaluate the studies in our final dataset.”
A systematic review of intergenerational co-residence between older people and adult children
Hanum, L., Newcombe, P. A., & Scott, T. (2024) · Journal of Family Studies · doi:10.1080/13229400.2024.2363785 · Cited by 26
“In addition, we also completed a quality assessment for every study we reviewed with the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2018).”
Ramli, A. H. M., Manaf, L. A., Zulkeflee, Z., & Andriyono, S. (2024) · Sustainable Production and Consumption · doi:10.1016/j.spc.2024.08.018 · Cited by 34
A scoping review of policies related to reducing energy drink consumption in children
Rostami, M., Babashahi, M., Ramezani, S., & Dastgerdizad, H. (2024) · BMC Public Health · doi:10.1186/s12889-024-19724-y · Cited by 41
Yosep, I., Fitria, N., Mardhiyah, A., Pahria, T., Yamin, A., & Hikmat, R. (2024) · BMC Nursing · doi:10.1186/s12912-024-02439-1 · Cited by 26
“In the mixed methods design, the authors used the Mixed Methods Appraisal Tool (MMAT) to conduct the quality appraisal assessment [21].”
Education for sustainable development (ESD) through participatory research (PR): A systematic review
Fernando, A. R., & Tajan, G. P. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.144237 · Cited by 26
Shin, Y., Park, E. J., & Lee, A. (2025) · Journal of Medical Internet Research · doi:10.2196/66442 · Cited by 52
“Quality assessment was conducted using the mixed methods appraisal tool [43] to ensure the methodological quality of the studies included (Multimedia Appendix 2 [44-61]).”
Tong, K., Thompson, C. W., Carin-Levy, G., Liddle, J., Morton, S., & Mead, G. E. (2025) · Age and Ageing · doi:10.1093/ageing/afaf084 · Cited by 44
Pyne, B., Asmara, O. J., & Morawska, A. (2025) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-025-00523-9 · Cited by 64
“The Mixed Method Appraisal Tool (MMAT; Hong et al., 2018), was engaged for its versatility in assessing studies of different designs.”
Public image of nursing: an integrative review of challenges and solutions
Moghbeli, G., Gardashkhani, S., & Soheili, A. (2025) · BMC Nursing · doi:10.1186/s12912-025-03160-3 · Cited by 26
“The quality appraisal process for this integrative review was conducted by two independent reviewers (GM and SG) using the Mixed Methods Appraisal Tool (MMAT) [25].”
Understanding the gaps in DR-TB care cascade in Nigeria: A sequential mixed-method study
Omenka, C. O., Boffa, J., Kuye, J., Dakum, P., Menzies, D., & Zarowsky, C. (2020) · Journal of Clinical Tuberculosis and Other Mycobacterial Diseases · doi:10.1016/j.jctube.2020.100193 · Cited by 28
“We used the Mixed Methods Appraisal Tool (MMAT) [25] to conceptualize, develop and interpret findings from this study [26].”
MethodBhochhibhoya, S., Dobbs, P. D., & Maness, S. B. (2020) · Preventive Medicine · doi:10.1016/j.ypmed.2020.106387 · Cited by 30
Bowman, R. L., Taylor, J., Muggleton, S., & Davis, D. L. (2021) · BMC Complementary Medicine and Therapies · doi:10.1186/s12906-021-03230-4 · Cited by 31
Fiaz, S., & Qureshi, M. A. (2021) · Future Business Journal · doi:10.1186/s43093-020-00046-5 · Cited by 27
Chan, K. K. L., & Shorey, S. (2021) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2021.12.002 · Cited by 37
Edgar, D., Moroney, T., & Wilson, V. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16232 · Cited by 26
Triece, P., Massazza, A., & Fuhr, D. C. (2022) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-022-02294-y · Cited by 30
A systematic review of autistic children’s prosocial behaviour
Ryan-Enright, T., O’Connor, R., Bramham, J., & Taylor, L. K. (2022) · Research in Autism · doi:10.1016/j.rasd.2022.102023 · Cited by 34
McCormack, S., Patel, K., & Smith, C. (2023) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.13143 · Cited by 28
Amsah, N., Isa, Z. M., Ahmad, N., & Manaf, M. R. A. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20054577 · Cited by 31
Ketosis and migraine: a systematic review of the literature and meta-analysis
Neri, L. D. C. L., Ferraris, C., Catalano, G., Guglielmetti, M., Pasca, L., Pezzotti, E., Carpani, A., & Tagliabue, A. (2023) · Frontiers in Nutrition · doi:10.3389/fnut.2023.1204700 · Cited by 28
“After selecting articles, the quality of evidence was checked using the Mixed Methods Appraisal Tool system (MMAT system) (27).”
MethodRyan, B. J., Clunne, S. M., Baker, C. J., Shiggins, C., Rose, M. L., & Kneebone, I. I. (2022) · Disability and Rehabilitation · doi:10.1080/09638288.2021.1925752 · Cited by 25
Woods, C., Kim, B., Guo, K., Nyguen, T., Taplayan, S., & Aronowitz, T. B. (2021) · Journal of the American Psychiatric Nurses Association · doi:10.1177/10783903211038050 · Cited by 29
“The mixed methods appraisal tool (MMAT) version 2018 for information professionals and researchers.”
MethodInfluentialDiane, R., Whiteing, N. L., & Aggar, C. (2022) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2022.07.002 · Cited by 25
Therapeutic Hospital Gardens: Literature Review and Working Definition
Nieberler-Walker, K., Desha, C., Bosman, C., Roiko, A. H., & Caldera, S. (2023) · HERD Health Environments Research & Design Journal · doi:10.1177/19375867231187154 · Cited by 44
“Based on Hong et al. (2018), the MMAT appraisal of the 24 primary research articles is presented in Supplementary Materials 2.1 and 2.2.”
BackgroundMethodInfluentialArmstrong, R. M., Schimke, E. A. E., Mathew, A., & Scarinci, N. (2023) · Language Speech and Hearing Services in Schools · doi:10.1044/2023_lshss-22-00168 · Cited by 36
Kapsali, M. Z., Livanis, E., Tsalikidis, C., Oikonomou, P., Voultsos, P. P., & Tsaroucha, A. (2024) · Cureus · doi:10.7759/cureus.54759 · Cited by 32
“…as the included articles employed not only normative ethics articles but also quantitative, qualitative, and mixed methods, the quality assessment of these articles was performed using the mixed methods appraisal tool (MMAT, version 2018), separately using the relevant sections of it [6].”
MethodFactors contributing to the promotion of moral competence in nursing
Wiisak, J., Stolt, M., Igoumenidis, M., Chiappinotto, S., Gastmans, C., Keogh, B., Mertens, E., Palese, A., Papastavrou, E., McCabe, C., et al. (2024) · Nursing Ethics · doi:10.1177/09697330241235305 · Cited by 39
Factors affecting family-centered care practice by nurses: A systematic review
Alqarawi, N., & Alhalal, E. (2024) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2024.06.008 · Cited by 25
Thoonsen, A. C., Schoten, S. M. V., Merten, H., van Beusekom, I., Schoonmade, L. J., Delnoij, D. M. J., & de Bruijne, M. C. (2024) · Health Policy · doi:10.1016/j.healthpol.2024.105135 · Cited by 37
Brückner, S., Sadare, O., Fesl, S., Scheibe, M., Lang, C., & Gilbert, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01568-4 · Cited by 33
“Quality assessment SB and OS independently assessed the quality of the included studies using the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Table 4) 56 Disagreements at any stage were resolved by discussion with SG.”
Applications of Artificial Intelligence in Psychiatry and Psychology Education: Scoping Review
Prégent, J., Chung, V.-H.-A., Adib, I. E., Désilets, M., & Hudon, A. (2025) · JMIR Medical Education · doi:10.2196/75238 · Cited by 44
Fuller, S. J., Philpot, U., & Working Group (2020) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.12731 · Cited by 26
Uhm, J., Choi, M., & Lee, H. (2020) · Nursing and Health Sciences · doi:10.1111/nhs.12786 · Cited by 31
Douglas, L., Donohue, G., & Morrissey, J. (2021) · Issues in Mental Health Nursing · doi:10.1080/01612840.2021.1978597 · Cited by 31
“Methodological quality of qualitative studies using MMat (Hong et al., 2018).”
MethodNasrawi, D., Latimer, S. L., Massey, D. L., & Gillespie, B. M. (2022) · Australian Critical Care · doi:10.1016/j.aucc.2022.01.007 · Cited by 24
Beliefs about food allergies in adolescents aged 11–19 years: A systematic review
Newman, K. L., Chater, A., & Knibb, R. C. (2022) · Clinical and Translational Allergy · doi:10.1002/clt2.12142 · Cited by 25
“17 The Mixed Methods Appraisal Tool (MMAT) 18 was used to assess the quality of studies to be included, and studies were excluded if they were assessed as poor quality.”
Racine, L., & Andsoy, İ. I. (2022) · Journal of Transcultural Nursing · doi:10.1177/10436596221085301 · Cited by 29
“We used the Critical Appraisal Skills Program (CASP) checklists (Critical Appraisal Skills Programme, 2020) and the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) to appraise the quality of the studies.”
BackgroundMethodInfluentialMoey, S. F., Sowtali, S. N., Ismail, M. F. M., Hashi, A. A., Azharuddin, N. S. M., & Mohamed, N. C. (2022) · Asian Pacific Journal of Cancer Prevention · doi:10.31557/apjcp.2022.23.12.3971 · Cited by 35
“This appraisal procedure was conducted using the Mixed-Method Appraisal Tool (MMAT) (Hong et al., 2018).”
MethodBrothers, T. D., Bonn, M., Lewer, D., Comeau, E. M., Kim, I., Webster, D., Hayward, A. C., & Harris, M. (2023) · Addiction · doi:10.1111/add.16257 · Cited by 34
Facial and Vocal Emotion Recognition in Adolescence: A Systematic Review
Zupan, B., & Eskritt, M. (2023) · Adolescent Research Review · doi:10.1007/s40894-023-00219-7 · Cited by 39
“The quality of included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2018).”
Direct Parent Engagement to Improve Fundamental Movement Skills in Children: A Systematic Review
Flynn, R. J., Pringle, A., & Roscoe, C. M. P. (2023) · Children · doi:10.3390/children10071247 · Cited by 50
“According to the MMAT, most of the studies were of excellent quality.”
BackgroundMethodInfluentialIdentifying the unmet supportive care needs of people affected by kidney cancer: a systematic review
O’Dea, A., Gedye, C. A., Jago, B., & Paterson, C. C. (2021) · Journal of Cancer Survivorship · doi:10.1007/s11764-021-01113-8 · Cited by 30
Richer, A. M. S., & Roddy, A. L. (2023) · The Journal of Mental Health Training, Education and Practice · doi:10.1108/jmhtep-07-2021-0088 · Cited by 25
Stephenson, Z., Johnson-Glauch, N., & Cruchley, S. (2025) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2025.2504621 · Cited by 25
“All studies were quality assessed using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) by the lead author.”
Hinneh, T. K., Jahn, A., & Agbozo, F. (2021) · International Health · doi:10.1093/inthealth/ihab054 · Cited by 32
Anakwenze, O. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.115124 · Cited by 34
Gambe, R. G., Clark, J., Meddick-Dyson, S., Ukoha-Kalu, B. O., Nyaaba, G. N., & Murtagh, F. E. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001785 · Cited by 28
“Two researchers (RG and BUK) conducted quality appraisal using the Mixed Methods Appraisal Tool (MMAT) [16].”
MethodInfluentialVeli‐Gold, S., Gilroy, J., Wright, W., Bulkeley, K., Jensen, H., Dew, A., & Lincoln, M. (2023) · Australian Journal of Rural Health · doi:10.1111/ajr.13011 · Cited by 24
de Campos, A. C., Hidalgo‐Robles, Á., Longo, E., Shrader, C., & Paleg, G. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15682 · Cited by 26
“22 Because F-words may not be explicitly addressed in studies, the study team developed the methods to identify the ingredients of the interventions and respective F-words through extensive discussions and consultation with experts.”
BackgroundNguyen, N. T., Kim, B., & Aronowitz, T. B. (2023) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12972 · Cited by 38
Sachan, T., Chhabra, D., & Abraham, B. (2023) · Games and Culture · doi:10.1177/15554120231201760 · Cited by 37
“Joanna Briggs Institute ’ s (2020) Checklist for Qualitative Research and Checklist for Analytical Cross-sectional Studies and the Mixed Methods Appraisal Tool (Hong et al., 2018) were utilized.”
Liu, M., Wang, J., Lou, J., Zhao, R., Deng, J., & Liu, Z. P. (2023) · Human Resources for Health · doi:10.1186/s12960-023-00874-w · Cited by 36
“Considering the varied types of the included studies (e.g., quantitative, qualitative, and mix-methods studies), we used the Mixed Methods Appraisal Tool (MMAT) version 2018 to evaluate the quality of the included studies [38].”
Method
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.