The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,101–2,200 of 4,578
Lehmann, K., & Leavey, G. (2023) · Clinical Child Psychology and Psychiatry · doi:10.1177/13591045231169093 · Cited by 5
de Leeuw, J. S. H., & Malcolm‐Smith, S. (2023) · Vulnerable Children and Youth Studies · doi:10.1080/17450128.2023.2224121 · Cited by 6
“Quality assessment of mixed-method studies was done using the Mixed Methods Appraisal Tool (MMAT), version 2018 (Hong et al., 2018).”
Issues Which Marginalize Females with ADHD - A Mixed Methods Systematic Review
Vincenti, S. C., Galea, M., & Briffa, V. (2023) · Humanities and Social Science Research · doi:10.30560/hssr.v6n2p30 · Cited by 7
“Studies which met inclusion criteria were appraised critically for methodological quality using the standardized instrument Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018). MMAT version 2018 was selected because it provides methodological quality criteria for quantitative, qualitative, and mixed m…”
MethodInfluentialHassani, M., Mate, K., TURCOTTE, R. E., Denis‐Larocque, G., Ghodsi, E., Tsimicalis, A., & Goulding, K. A. (2023) · Journal of Surgical Oncology · doi:10.1002/jso.27390 · Cited by 5
What strategies can motivate businesses to consider implementing ESG?
Gold, J. (2023) · Muma Business Review · doi:10.28945/5207 · Cited by 5
Hanff, A.-M., Pauly, C., Pauly, L., Rauschenberger, A., Leist, A. K., Krüger, R., Zeegers, M. P., & McCrum, C. L. (2023) · Gait & Posture · doi:10.1016/j.gaitpost.2023.11.013 · Cited by 7
Bahari, N. I., Sutan, R., & Mahdy, Z. A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297563 · Cited by 5
Ma, Y., Law, K., Hassan, L., Nenadić, G., & van der Veer, S. N. (2024) · JMIR Pediatrics and Parenting · doi:10.2196/56919 · Cited by 4
Myers, L. S., Johnston, E. A., Zajdlewicz, L., Viljoen, B., Kelly, S., Perry, N., Stiller, A., Crawford-Williams, F. M., Chan, R. J., Emery, J. D., et al. (2024) · Psycho-Oncology · doi:10.1002/pon.6321 · Cited by 4
Quantitative Metrics in Mass-Gathering Studies: A Comprehensive Systematic Review
Çalışkan, C., Kuday, A. D., Özcan, T., et al. (2024) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x2400027x · Cited by 5
Panda, A., Parida, J., Jena, S., Pradhan, A., Pati, S., Kaur, H., & Acharya, S. K. (2024) · Journal of Family Medicine and Primary Care · doi:10.4103/jfmpc.jfmpc_1674_22 · Cited by 7
Paramedicine research with family and bystanders: A methodological review
Satchell, E., Anderson, N. E., & Gott, M. (2024) · Paramedicine · doi:10.1177/27536386241255387 · Cited by 5
“45 All three quantitative studies satis fi ed all elements of their corresponding MMAT categories, indicating high quality.”
InfluentialLegitimation Code Theory's role in shaping nursing education: An integrative review
Bowdler, S., Nielsen, W., Meedya, S., Matthews, A., & Salamonson, Y. (2024) · Nurse Education Today · doi:10.1016/j.nedt.2024.106269 · Cited by 7
“The Mixed Methods Appraisal Tool (MATT) was used to assess the quality of the studies (Hong et al., 2018).”
da Luz, E. M. F., Munhoz, O. L., Greco, P. B. T., et al. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7048.4176 · Cited by 5
Interventions to promote medication adherence among children with epilepsy: An integrative review
Kangwal, C., Thato, R., Ua-Kit, N., & Visudtibhan, A. (2024) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2024.06.015 · Cited by 5
Elder Abuse in Older Adults With Dementia: Protective Factors and Adverse Effects
Wei, W., Balser, S., Nguyen, A. W., & Qin, W. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241265379 · Cited by 6
“The MMAT is an essential instrument designed for evaluating studies within the context of systematic mixed studies reviews, which encompass qualitative, quantitative, and mixed-methods research.”
InfluentialGeleto, A. B., Vasilevski, V., & Sweet, L. P. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02628-8 · Cited by 8
“…qualitative and quantitative components were adequately interpreted, (iv) whether divergences and inconsistencies between quantitative and qualitative results were adequately addressed, and (v) whether different components of the study adhered to the quality criteria of each method involved [29].”
BackgroundMethodKaraba, F., Nunes, A. R., Geddes, O., Atherton, H., Dahlmann, F., Eccles, A., Gregg, M., Spencer, R. A., Twohig, H., & Dale, J. R. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-087795 · Cited by 6
“36”
The battle against screen time: a systematic review of hedonic information systems use reduction
Nascimento, P., Oliveira, T., & Neves, J. (2024) · Internet Research · doi:10.1108/intr-10-2023-0857 · Cited by 7
Sanantonio, A. G., & Botija, M. (2024) · Health & Social Care in the Community · doi:10.1155/2024/3010747 · Cited by 8
Jostock, C., Forde, H., Roberts, N. W., Jebb, S. A., Pechey, R., & Bandy, L. (2025) · Public Health Nutrition · doi:10.1017/s1368980025000035 · Cited by 5
“Study quality was rated independently by one reviewer and verified by a second reviewer (LB or CJ) using the MMAT (26) .”
InfluentialCiemer, C., Kröper, L., Klotzbier, T. J., Ghellal, S., & Schott, N. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1489258 · Cited by 5
Fouskas, T. (2025) · International perspectives on equality, diversity and inclusion · doi:10.1108/s2051-233320250000012019 · Cited by 6
Zabeen, S., Phua, D., Mohammadi, L., & Lawn, S. (2020) · Journal of Mental Health · doi:10.1080/09638237.2020.1818194 · Cited by 7
“The Mixed Methods Appraisal Tool (MMAT) was used to appraise the quality of the selected articles, given its ability to assess all types of empirical studies including qualitative and mixed methods studies (Hong et al., 2018).”
MethodShon, S., Jeon, H., & Hwang, H. (2021) · Child Health Nursing Research · doi:10.4094/chnr.2021.27.2.111 · Cited by 5
“The Mixed Methods Appraisal Tool [13] was applied for a critical quality appraisal of the analyzed studies.”
MethodD’Errico, D., Schroder, T. A., & Gresswell, D. M. (2022) · Psycho-Oncology · doi:10.1002/pon.5964 · Cited by 6
Diagnostic delay of myositis: protocol for an integrated systematic review
Namsrai, T., Desborough, J., Chalmers, A., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060312 · Cited by 5
Ability of management of families of children with chronic conditions for care at home
Felizardo, M. J. D. A., da Silva, J. B., Neves, E. T., et al. (2022) · Escola Anna Nery · doi:10.1590/2177-9465-ean-2022-0071en · Cited by 5
“15 The theoretical framework of the Family Management Style was used 14 as the theoretical framework of the study and the guidelines of the Mixed Methods Appraisal Tools 16 for the preparation of the research report and the article.”
MethodManagement of overweight and obesity in children and adolescents by nurses: a mixed-method study
Oliveira, R. C., Souto, R. Q., dos Santos, J. L. G., Reichert, A. P. D. S., Ramalho, E. L. R., & Collet, N. (2022) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6294.3789 · Cited by 5
“In addition to these guides of the EQUATOR Network, the recommendations of the Mixed Methods Appraisal Tool (MMAT)(22) were adopted for the methodological rigor of mixed studies.”
MethodSandberg, K., Olsson, C., Gjevjon, E. R., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-064610 · Cited by 5
Thorne, T. E., Duan, Y., Slubik, S., & Estabrooks, C. A. (2023) · JMIR Research Protocols · doi:10.2196/40390 · Cited by 5
“…Studies, Checklist for Cohort Studies, Checklist for Case-Control Studies, Checklist for Case Series, Checklist for Prevalence Studies, Checklist for Analytical Cross-Sectional Studies, Checklist for Qualitative Research) [41] and the Mixed Methods Appraisal Tool, version 2018 [42].”
Ordóñez, C. (2023) · Urban forestry & urban greening · doi:10.1016/j.ufug.2023.127921 · Cited by 6
Thomas, N., Atherton, H., Dale, J. R., et al. (2023) · BJGP Open · doi:10.3399/bjgpo.2023.0010 · Cited by 5
“The MMAT (version 2018) 19 was used to conduct quality assessment of all studies fitting the inclusion criteria.”
MethodHow to develop young physical activity leaders? A Delphi study
Steward, I. P., Young, E., Dogra, S. A., Stamp, E., Daly-Smith, A., Siddique, K., Morgan, K., Crowther, J., & Hall, J. (2023) · PLoS ONE · doi:10.1371/journal.pone.0286920 · Cited by 6
A Systematic Review of the Reminiscence Functions Scale and Implications for Use with Older Adults
Marques, C., Dias, S. F., & de Sousa, L. X. M. (2023) · Clinical Gerontologist · doi:10.1080/07317115.2023.2274989 · Cited by 8
“All articles were included for data extraction, since excluding studies with low methodological quality is discouraged to avoid losing relevant information in the systematic review (Hong et al., 2018; Pace et al., 2012).”
Rajendran, P., Jarašiūnaitė–Fedosejeva, G., İşbir, G. G., & Shorey, S. (2023) · Palliative Medicine · doi:10.1177/02692163231217597 · Cited by 12
“25,26 Any disagreements that arose were resolved through discussions to consensus.”
BackgroundSherborne, V., Ejegi‐Memeh, S., Tod, A., Taylor, B. H., Hargreaves, S., & Gardiner, C. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-075071 · Cited by 6
Ranabhat, C. L., & Acharya, D. (2020) · Archives of Public Health · doi:10.1186/s13690-020-00418-x · Cited by 8
Thompson, J., Fairbrother, H., Spencer, G., et al. (2019) · Global Health Promotion · doi:10.1177/1757975919888452 · Cited by 5
Meta-Analytical Evidence on Mental Disorder Symptoms During the COVID-19 Pandemic in Latin AmericaPreprint
Zhang, S. X., Batra, K., Liu, T., Dong, R. K., Xu, W., Yin, A., Delios, A., Chen, B. Z., Chen, R. Z., Miller, S., et al. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3858820 · Cited by 5
Hlongwa, M., Hlongwana, K. W., Makhunga, S., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-1311742/v1 · Cited by 5
A systematic review of multi-modal prehabilitation in breast cancerPreprint
Toohey, K. L., Hunter, M., McKinnon, K., Casey, T., Turner, M. R., Taylor, S., & Paterson, C. C. (2022) · Research Square · doi:10.21203/rs.3.rs-1976805/v1 · Cited by 5
Lessard-Deschênes, C., Goulet, M.-H., & Pariseau-Legault, P. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-065393 · Cited by 4
Lo, U. C., Musa, H., Li, J., et al. (2022) · Palliative & Supportive Care · doi:10.1017/s1478951522001547 · Cited by 4
Chow, H. Y., Salleh, N. S., Cheng, K. W., & Totsika, V. (2024) · International Journal of Developmental Disabilities · doi:10.1080/20473869.2024.2331815 · Cited by 6
“The risk of bias assessment was conducted using the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al. 2018).”
Game performance assessment instrument in physical education: a systematic review from 2015 to 2024
Manso-Lorenzo, V., Evangelio, C., Tendero, G. R., & González‐Víllora, S. (2024) · Physical Education and Sport Pedagogy · doi:10.1080/17408989.2024.2432313 · Cited by 12
Servant Leadership Style in High Education Institutions: A Systematic Literature Review
Dul, V., Sam, R., Hak, S., Bou, D., Vy, S., & Kheuy, S. (2024) · European Journal of Contemporary Education and E-Learning · doi:10.59324/ejceel.2024.2(6).08 · Cited by 13
Doody, O., Graham, M., Hegarty, O., Sloane, A. M., Walsh, P., Synnott, M., Russell, M., Dunworth, T., Sheridan, J., & Murphy, L. (2025) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2025.100304 · Cited by 6
“…were extracted into a data extraction table (Table 2), and each paper was assessed for quality and rigour utilising the Mixed Methods Appraisal Tool (Hong et al., 2018)) and the Authority, Accuracy, Coverage, Objectivity, Date, Significance checklist for appraising grey literature (Tyndall, 2010).”
Belachew, S. A., Bizuayehu, H. M., Diaz, A., Jahan, S., Crengle, S. M., Fong, K. M., & Garvey, G. (2025) · Health Promotion Journal of Australia · doi:10.1002/hpja.70001 · Cited by 5
“The included articles were independently appraised by two reviewers (SAB and HBM) using the Mixed Methods Appraisal Tool (MMAT) [25], designed to appraise reviews that include different study designs: qualitative, quantitative and mixed methods studies (Table S2).”
Karhu, V.-M., Peltonen, A., & Flykt, M. S. (2026) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2026.108074 · Cited by 4
Burnout and work engagement in advanced practice cancer nurses: A scoping review
Mudiyanselage, A. C. R., Bennett, P. N., Coyne, E., Hughes, L., Tapsall, D., & Forster, E. (2025) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2025.102860 · Cited by 5
What Is a Good Death in South Asia? A Systematic Review and Narrative Synthesis
Wijeyaratne, L., Spruijt, O., Jayasinghe, S., Kane, S. S., Ramadasa, U., & Philip, J. A. M. (2025) · Journal of Nursing Scholarship · doi:10.1111/jnu.70002 · Cited by 6
“The Critical Appraisal Skills Program (CASP) tool was used to appraise qualitative studies (Critical Appraisal Skills Programme (2022), CASP Qualitative Studies Checklist [online]) and the Mixed Method Assessment Tool (MMAT) (Quan Nha Hong et al. 2018) for mixed-method studies.”
Ecological Momentary Assessment to Measure Social Connectedness in Older Adults: Integrative Review
Choi, S., Kang, H., Shin, J., Chu, S. H., & Choi, J. (2025) · Journal of Medical Internet Research · doi:10.2196/66324 · Cited by 5
“Two authors, SC and HK, independently assessed the quality of the selected studies using the 2018 version of the Mixed-Methods Appraisal Tool (MMAT) [25].”
Balcha, A. R., & Chen, H. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2492297 · Cited by 6
“Additionally, the quality of selected articles was assessed using the Mixed Methods Appraisal Tool (MMAT) developed by Hong et al. (2018).”
Unpacking the Player: A review of consumption behaviours beyond gameplay in esports
Majiri, N. S. B., Othman, M. K., & Zulkiply, N. B. (2025) · Entertainment Computing · doi:10.1016/j.entcom.2025.100966 · Cited by 4
Keogh, A. J., & McGillicuddy, D. (2025) · Reading Research Quarterly · doi:10.1002/rrq.70017 · Cited by 20
“As this review endeavored to include qualitative, quantitative, and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT) (Hong et al. 2018) was utilized to assess the methodological quality of all included studies due to its inter-rater reliability, usability, and content validity (Pace et al. 2012; Pluye 20…”
InfluentialKirwan, L., Apriyanti, E., Sheaf, G., DeVries, J., & Coyne, I. (2025) · Journal of Clinical Nursing · doi:10.1111/jocn.17861 · Cited by 5
“The quality of included studies was independently assessed by the first author (L.K.) using the Mixed Methods Appraisal Tool (MMAT) (Pluye 2012; Hong et al. 2018).”
Muzdalifah, F., Kurniawati, F., Mangunsong, F., Setiamurti, A., Levyadi, J., Bracken, S., & Martono, W. (2025) · Innovations in Education & Teaching International · doi:10.1080/14703297.2025.2513540 · Cited by 3
“The MMAT was selected due to its established psychometric properties and applicability to qualitative, quantitative, and mixed-methods research (Hong et al., 2018).”
Aggarwal, M., Bourgeault, I. L., Dalo, S., Kokorelias, K. M., Greenberg, L. S., Kreutzweiser, B., Samson, K., Sorensen, L., Kemp, C., Schieck, D., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0328708 · Cited by 5
“The MMAT is deemed suitable for this review because it is specifically designed to assess the methodological quality of various types of studies, including qualitative, quantitative, and mixed-methods research [80].”
Maleki, M., Mardani, A., Kakemam, E., Huertas‐Zurriaga, A., & Vaismoradi, M. (2025) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2025.2537190 · Cited by 8
“The quality assessment of the included studies was conducted using the 2018 version of the Mixed Methods Appraisal Tool (MMAT) [39].”
Comber, E. R., August, D. L., Nguyen, L. N., de Souza, S., Ullman, A. J., Byrnes, J., & Keogh, S. J. (2025) · Hospital Pediatrics · doi:10.1542/hpeds.2024-008248 · Cited by 5
Utilization of telerehabilitation in TKR patients: A systematic review
Salehian, F., Alipour, J., & Zakerabasali, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0324074 · Cited by 8
“MMAT is an appropriate tool for evaluating all study designs, which decreases the risk of bias [9].”
Alter, E., Fényes, H., & Pusztai, G. (2025) · Review of Education · doi:10.1002/rev3.70091 · Cited by 8
Beyond apologies: A systematic review of forgiveness in different relationship contexts
Lahiri, A., & Padmakumari, P. (2025) · Current Psychology · doi:10.1007/s12144-025-08246-4 · Cited by 16
“The mixed method study was assessed utilizing the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2018 ).”
Sanchez, C. G., Kharko, A., Hägglund, M., Riggare, S. S., & Blease, C. (2025) · JMIR Research Protocols · doi:10.2196/73602 · Cited by 4
Luo, X., Rose, R. A. C., & Awang, A. (2025) · Frontiers in Sustainable Cities · doi:10.3389/frsc.2025.1628137 · Cited by 5
Shared decision-making for early-stage breast cancer treatment: An evolutionary concept analysis
Chae, H., Ji, H.-J., Moon, H., Lee, Y., & Son, Y. (2025) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2025.100775 · Cited by 4
Penfold, S. M., Cunningham, J. A., Whelan, P., McCabe, M. G., & Ainsworth, J. D. (2025) · Pathophysiology · doi:10.3390/pathophysiology32030046 · Cited by 8
“To assess the quality of each of the studies methodology, the lead researcher used the Mixed-Method Appraisal Tool [24] to critically evaluate the different quantitative study designs that have been included in this review.”
Li, Y., Chang, F., Zhang, W., Ren, Z., Chen, Y., & Liu, Z. (2025) · Journal of Medical Internet Research · doi:10.2196/66981 · Cited by 8
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [69,70].”
Chivasa, T., Nunu, W. N., Dhlamini, M., Maviza, A., & Ndagurwa, G. N. (2025) · Health Services Insights · doi:10.1177/11786329251374245 · Cited by 7
“26 The tool was applied across all included studies, irrespective of the study design, except for gray literature, which was assessed using a separate appraisal checklist.”
Laser-Assisted Drug Delivery for Hypertrophic Scar Treatment: A Scoping Review
Shilova, M., Plummer, K., Ware, R. S., Kimble, R. M., Clark, J. M., Cho, E., Lucinda, M., Kimble, L., Meikle, B., Lauren, K., et al. (2025) · Journal of Burn Care & Research · doi:10.1093/jbcr/iraf167 · Cited by 4
“32 The MMAT was designed to be used for literature reviews that include multiple study designs and can be used to assess 5 types of studies.”
Ding, Y., Wan, Q., Kim, J.-C., Liu, W., & Ji, W. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1681184 · Cited by 7
“The MMAT is a widely used tool that facilitates the quality assessment of studies with diverse designs (Hong et al., 2018).”
Endometriosis Support and Development of Digital Technology–Based Interventions: Systematic Review
Pavic, T., Nadarajah, K., Somat, A., Cabagno, G., & Terrade, F. (2025) · JMIR Human Factors · doi:10.2196/71859 · Cited by 19
“The risk of bias of each study included in this review was assessed with the Mixed Methods Assessment Tool (MMAT) [47].”
Bhatia, R. K., Bhardwaj, P., Chaudhry, A., Ghosh, A., Parmar, A., Ahluwalia, Y., Singh, G., Prasad, D., Verma, P., Dahiya, N., et al. (2025) · Digital Health · doi:10.1177/20552076251387056 · Cited by 8
“20 Additionally, the quality of the systematic reviews included in this study was appraised using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2).”
Nurse-led fall prevention programs in acute care settings: An integrative review
Al-Ghareeb, S. A., AlOtaibi, N. G., Sallam, L. A., & Innab, A. (2025) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2025.100440 · Cited by 10
“Two independent reviewers critically appraised the studies using the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2018).”
Badibanga, D. T., Mushage, O. B., Manirabona, A., et al. (2025) · Advances in Internet of Things · doi:10.4236/ait.2025.154006 · Cited by 4
McKendrick-Calder, L., Smith, S., Choate, J., & Nelson, J. (2025) · Higher Education Research & Development · doi:10.1080/07294360.2025.2559649 · Cited by 6
“Study appraisals were undertaken using the Mixed Method Appraisal Tool (Hong et al., 2018) for quantitative studies and the Joanna Briggs tool for qualitative studies (Lockwood et al., 2015).”
Long-term athlete development in schools: a systematic review and narrative synthesis
Söker, N., Schamerowski, J., Zech, A., Steib, S., & Rahlf, A. L. (2025) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2025.1706060 · Cited by 16
Jafarimanesh, H., Ghiyasvandian, S., Lyman, B., Ghanaati, H., & Zakerimoghadam, M. (2025) · Teaching and learning in nursing · doi:10.1016/j.teln.2025.10.013 · Cited by 5
Mobile crisis effectiveness: a systematic review and associated functions and forms framework
Peters, R. L., Fine, J., Newton, H., Dalack, G. W., Watson, A. C., & Lich, K. H. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13806-2 · Cited by 8
“The quality of all studies was assessed using the Mixed Method Appraisal Tool (MMAT) [41], which was deemed to be appropriate for this study because both quantitative and qualitative studies were included.”
The longitudinal effects of social media on sleep among youth: A scoping review
Sullivan, E. C., Dubicka, B., Kirsop, H., & Henderson, L. M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100908 · Cited by 9
Policing as a determinant of transgender health: a critical narrative review
Abidin, M. Z., & Husin, A. (2025) · International Journal of Transgender Health · doi:10.1080/26895269.2025.2586758 · Cited by 6
Chen, C. K. (2026) · International Journal of Lean Six Sigma · doi:10.1108/ijlss-06-2025-0161 · Cited by 12
Russotto, S., Conti, A. A., Masini, A., Valenta, S. T., Vanhaecht, K., Mira, J. J., & Panella, M. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1703768 · Cited by 8
“Speci fi cally, the risk of bias of the included studies was evaluated using JBI checklists for qualitative studies (12), case series (13), quasi-experimental study (14), narrative (15), prevalence (16), and cross-sectional studies (17), as well as the MMAT tool for mixed-methods studies (18).”
Zhong, Y., Guo, K., Xu, L., & Fryer, L. K. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100767 · Cited by 6
Baschab, J. F., Moehler, E., & Hussong, J. (2026) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-025-02945-w · Cited by 9
“As this review included both randomized controlled trial (RCT) and nonrandomized controlled studies, the Mixed Methods Appraisal Tool (MMAT) was used to assess the methodological quality of the included studies [52 ].”
Meißner, N., Ramos‐Pascual, S., Ortwig, K. A., van Rooij, F., Schrednitzki, D., Stöve, J., & Halder, A. M. (2026) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70651 · Cited by 7
“Methodological quality of the eligible studies was independently assessed by two researchers (L.N. and M.H.) using the Mixed Methods Appraisal Tool (MMAT) [27, 28].”
Huang, M., Mansor, A. N., & Kamaruzaman, F. M. (2026) · Cogent Business & Management · doi:10.1080/23311975.2026.2621436 · Cited by 10
Xu, T., Liu, C., Li, L., Song, D., Rose, G. M., & Zhu, S. (2026) · Journal of Palliative Medicine · doi:10.1177/10966218261418542 · Cited by 8
Machkour, B., Rouky, N., & Abriane, A. (2026) · Multimodal Transportation · doi:10.1016/j.multra.2026.100298 · Cited by 8
Chimbo, B., Maguraushe, K., & Ndlovu, B. M. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1761456 · Cited by 14
Hashad, N., Jabbari, S., Fahmy, A. M., et al. (2026) · International Journal of Pharmacy Practice · doi:10.1093/ijpp/riag010 · Cited by 4
Buthelezi, L., Maseko, M. S. T., & Downs, C. T. (2026) · Biodiversity and Conservation · doi:10.1007/s10531-026-03275-2 · Cited by 8
Nawaz, I., Cuenen, A., Ahmed, T., Zeb, A., Wets, G., & Janssens, D. (2026) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2026.103591 · Cited by 8
Ameen, F. (2026) · Smart Learning Environments · doi:10.1186/s40561-026-00447-z · Cited by 11
“The quality assessment process followed an objective protocol in line with the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
Uygun, E., & Demir, B. (2026) · Discover Education · doi:10.1007/s44217-026-01504-y · Cited by 24
“Methodological quality was appraised using the Mixed Methods Appraisal Tool (MMAT) [34], selected due to the inclusion of qualitative, quantitative, and mixed-methods studies.”
Shi, H., Huang, J., Jin, Y., & Han, M. (2026) · Journal of Medical Internet Research · doi:10.2196/80581 · Cited by 10
“The MMAT User Manual mentions the treatment of low-quality studies, which are rounded off according to the topic of the study; in this study, these 2 papers were valuable in contributing to the research topic and were therefore not excluded [25].”
Killeen, O. J., Saylor, K., Hogan, C. M., Jacobson, A., Collins, M. E., & Ehrlich, J. R. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-22-00026 · Cited by 5
Nurses’ professional competencies in urgency and emergency units: A mixed-methods study
Ferreira, K. M., Balsanelli, A. P., & dos Santos, J. L. G. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6554.3936 · Cited by 4
Systematic Review of Time Management Practices Among Nurse Managers
Qtait, M. T. (2023) · Africa Journal of Nursing and Midwifery · doi:10.25159/2520-5293/13507 · Cited by 13
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.