The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,201–1,300 of 4,578
Beres, L. K., Underwood, A., Tourneau, N. L., Kemp, C. G., Kore, G., Yaeger, L., Li, J., Aaron, A., Keene, C., Mallela, D. P., et al. (2024) · Journal of the International AIDS Society · doi:10.1002/jia2.26258 · Cited by 26
Chen, L., Sleeman, K. E., Bradshaw, A., Sakharang, W., Mo, Y., & Ellis-Smith, C. I. (2024) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2024.105036 · Cited by 17
O’Connor, G., Reis, K., Desha, C., & Burkett, I. (2024) · Agriculture and Human Values · doi:10.1007/s10460-024-10601-3 · Cited by 25
“The MMAT process involved measuring the quality of the article’s research questions, if chosen methods, measurements, participant selection and data analysis were appropriate to address those questions, and if findings/ outcomes adequately addressed these questions (Hong et al. 2018).”
Idris, N. D. M., Zulkepli, N. A., Ern, L. K., & Zainol, M. R. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.122045 · Cited by 26
Jensen, T. M. (2024) · Journal of Family Theory & Review · doi:10.1111/jftr.12581 · Cited by 20
Growth mindsets in academics and academia: a review of influence and interventions
Sousa, B. J., & Clark, A. M. (2024) · Journal of Higher Education Policy and Management · doi:10.1080/1360080x.2024.2384003 · Cited by 23
“The quality of each included study was appraised using the appropriate quality appraisal tool (CASP, 2020; Hong et al., 2018).”
Clinical Simulation in Nursing Students’ Safe Medication Administration: A Systematic Review
Shahzeydi, A., Dianati, M., & Kalhor, F. (2024) · Iranian Journal of Nursing and Midwifery Research · doi:10.4103/ijnmr.ijnmr_323_23 · Cited by 25
“Additionally, three articles were excluded based on qualitative evaluation using CONSORT 2010 Statement, [28] the Mixed Methods Appraisal Tool (MMAT), [29] and JBI Critical Appraisal Checklist for Quasi‑Experimental Studies.”
MethodChang, F., Wang, G., & Gu, Z. (2024) · Innovation in Aging · doi:10.1093/geroni/igae077 · Cited by 18
“The quality of studies was evaluated by the two authors (FC and GW) using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) (Supplementary Table 3).”
MethodLesbian and Gay Population, Work Experience, and Well-Being: A Ten-Year Systematic Review
Lacatena, M., Ramaglia, F., Vallone, F., Zurlo, M. C., & Sommantico, M. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21101355 · Cited by 21
“The Mixed-Methods Appraisal Tool (MMAT) version 2018 [22] was used to assign the quality rating.”
BackgroundMethodRanken, E., Wyse, D., Manyukhina, Y., & Bradbury, A. (2024) · The Curriculum Journal · doi:10.1002/curj.304 · Cited by 40
Children’s Pathways into Commercial Sexual Exploitation in the United States: A Systematic Review
Godoy, S. M., Hall, W. J., Cal, T., Bath, E. P. J., Abrams, L. S., Goode, R. W., Jensen, T. M., & Chapman, M. V. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241297420 · Cited by 19
“Methodological quality of identified studies was appraised using relevant criteria from the Mixed Methods Appraisal Tool (MMAT), version 2018 (Hong et al., 2018).”
Sun, M., Kraus, T., Pauli, R., & Garus, C. (2025) · Wellbeing Space and Society · doi:10.1016/j.wss.2025.100236 · Cited by 26
Lerdsuwanrut, N., Zamani, R., & Akrami, M. (2025) · Sustainability · doi:10.3390/su17020491 · Cited by 17
Mertens, J. F., Hessel, M. H. M., Kempen, T. G. H., Böhringer, S., Norbart, A. F., Walinga, C. W., & van Gelder, T. (2025) · BMC Medical Education · doi:10.1186/s12909-024-06574-w · Cited by 22
Miguel, P. A. S., Álvarez-Sánchez, J. L., Alcalá, D. H., Pueyo, Á. P., & Tapia‐Serrano, M. Á. (2025) · Quest · doi:10.1080/00336297.2024.2443937 · Cited by 23
“The methodological quality of studies was measured using the Mixed Method Appraisal Tool (Hong et al., 2018).”
Factors influencing vaccine hesitancy toward non-covid vaccines in South Asia: a systematic review
Stuetzle, S. C. W., Willis, M. D., Barnowska, E. J., Bonkass, A.-K., & Fastenau, A. (2025) · BMC Public Health · doi:10.1186/s12889-025-22462-4 · Cited by 23
“For mixed-method studies the Mixed Method Appraisal Tool (MMAT) [30] was used, for qualitative studies the NICE checklist [31], and for quantitative studies the Critical Appraisal Toolkit (CAT) [32].”
Implementing remote triage in large health systems: A qualitative evidence synthesis
Lewinski, A. A., Rushton, S., Voorhees, E. E. V., Boggan, J. C., Whited, J. D., Shoup, J. P., Tabriz, A. A., Adam, S. S., Fulton, J. J., Gordon, A. M., et al. (2020) · Research in Nursing & Health · doi:10.1002/nur.22093 · Cited by 15
Health‐related guilt in chronic primary pain: A systematic review of evidence
Serbic, D., Evangeli, M., Probyn, K., & Pincus, T. (2021) · British Journal of Health Psychology · doi:10.1111/bjhp.12529 · Cited by 17
Botham, J., Shilling, V. M., & Jones, J. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0063 · Cited by 18
Evidence of religious/spiritual singing and movement in mental health: A systematic review
Malviya, S., Zupan, B., & Meredith, P. J. (2022) · Complementary Therapies in Clinical Practice · doi:10.1016/j.ctcp.2022.101567 · Cited by 18
Grady, A., Jackson, J. K., Lum, M., Delaney, T. M., Jones, J. Z., Kerr, J., Falkiner, M., & Yoong, S. L. (2022) · Preventive Medicine · doi:10.1016/j.ypmed.2022.107011 · Cited by 19
Dwyer, L., Dowding, D., & Kearney, R. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060223 · Cited by 14
Gelineau, A., Perrochon, A., Daviet, J., & Mandigout, S. (2022) · Journal of Rehabilitation Medicine · doi:10.2340/jrm.v54.2270 · Cited by 20
“Variability in acceptable discontinued intervention/ drop-out rates was found in the literature, ranging from 5%, to 20%, and 30% for follow-up of more than 1 year (46).”
BackgroundVidela, C. E. C., & Melillo, R. C. C. (2023) · Salud Mental · doi:10.17711/sm.0185-3325.2023.005 · Cited by 24
“A similar gap in terms of the facilitator’s expertise is found in Jasbi et al. (2018), with CBT therapists and MBSR instructors carrying out a differ - ent modality of mindfulness, MBCT.”
InfluentialAbdi, F. S., Aghajani, A., Hemmati, S., et al. (2023) · Indian Journal of Ophthalmology · doi:10.4103/ijo.ijo_643_22 · Cited by 14
Mulqueeny, D. M., Nkabini, S. M., & Pokiya, M. H. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01604-w · Cited by 15
“included evidence is appropriate, this review will utilise the mixed-method appraisal tool (MMAT) version 2018 to appraise the quality of all included evidence [49].”
MethodCleaver, K., Essex, R., Malamateniou, C., Narramore, N., Shekede, H., Vargo, E. J., & Weldon, S. M. (2021) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2020.11.008 · Cited by 15
Boss, M., Saxby, N., Pritchard, D. A., Pérez‐Escamilla, R., & Clifford, R. M. (2021) · Maternal and Child Nutrition · doi:10.1111/mcn.13160 · Cited by 15
Park, M., Pawliuk, C., Nguyen, T., Griffitt, A., Dix‐Cooper, L., Fourik, N., & Dawes, M. G. (2021) · Eurosurveillance · doi:10.2807/1560-7917.es.2021.26.14.2001506 · Cited by 20
“Assessment of study quality An adaptation of the Mixed Methods Appraisal Tool was used to assess study quality [19].”
MethodYarwood, B., Taylor, R., & Angelakis, I. (2024) · Community Mental Health Journal · doi:10.1007/s10597-023-01196-w · Cited by 14
“Quality of the studies was assessed using the Mixed Meth-ods Appraisal Tool (MMAT) which is based on the following five questions (Hong et al., 2018): (i) Is the approach appropriate to the research question?”
Neri, L. D. C. L., Mariotti, F., Guglielmetti, M., Fiorini, S., Tagliabue, A., & Ferraris, C. (2024) · Frontiers in Nutrition · doi:10.3389/fnut.2024.1250683 · Cited by 13
Management of urinary incontinence in females by primary care providers: a systematic review
Charette, M., Pacheco‐Brousseau, L., Poitras, S., Ashton, R., & McLean, L. (2023) · British Journal of Urology · doi:10.1111/bju.16234 · Cited by 16
“Risk of bias assessment for included studies was carried out using the Mixed Methods Appraisal Tool (MMAT) [23,24].”
MethodArmour, R., Nielsen, S., Buxton, J. A., Bolster, J. L., Han, M. X., & Ross, L. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.11.031 · Cited by 12
Phillips, N., Maréchal, L., Ventura, B. A., & Cooper, J. J. (2024) · Animal Behavior and Cognition · doi:10.26451/abc.11.04.05.2024 · Cited by 19
“The Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018) was used for quality assessment (Supplementary material S3) to inform analysis and interpretation of results, not to remove studies perceived to be poor.”
Interventions and Programs Using Native Foods to Promote Health: A Scoping Review
Lopes, C. V. A., Neri, J. L. D. S. A., Hunter, J. F. M., Ronto, R., & Mihrshahi, S. (2024) · Nutrients · doi:10.3390/nu16234222 · Cited by 25
“The tool uses criteria related to the clarity of the research questions, appropriateness of the study design, representativeness of the sample, outcome data’s completeness, adherence to the intervention, and interpretation of the results [38].”
Nikolovski, J., Kim, B., Morton, R. L., Mercieca‐Bebber, R., Lévesque, J., Tinsley, M., Sutherland, K., Rossiter, B., Fagan, M., Hartas, G., et al. (2025) · Quality of Life Research · doi:10.1007/s11136-025-03913-2 · Cited by 21
“To appraise the quality of mixed-method studies the Mixed Methods Appraisal Tool (MMAT) was used as it was developed for mixed-method systematic reviews [33].”
What are the experiences of nurses returning to work following maternity leave: a scoping review
Johnson, E., Elder, E. G., & Kosiol, J. (2025) · BMC Nursing · doi:10.1186/s12912-024-02625-1 · Cited by 16
“The MMAT tool is designed to critically appraise the quality of qualitative research, quantitative research, mixed methods studies, randomised controlled trails, and non-randomised studies, through use of specific methodological quality criteria [19].”
Qi, M., Shen, X., Zeng, Y., Lin, X., Suliman, M., & Li, P. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106104 · Cited by 16
Leiviska, E., Pezaro, S., Kneafsey, R., Morini, L., & DeWinter, A. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106701 · Cited by 21
Foster, K., Alexander, L., Steele, M. C., Hale, T., Hutchison, L., Trejo, R., & Boardman, J. (2025) · BMC Nursing · doi:10.1186/s12912-025-03066-0 · Cited by 16
“The study is reported using The Mixed Methods Appraisal Tool [28].”
Çankaya, Ö., Martin, M., & Haugen, D. (2025) · Journal of Intelligence · doi:10.3390/jintelligence13050052 · Cited by 45
“The MMAT starts with two initial questions consistent across all study methods, followed by five specific questions based on the study’s design (i.e., qualitative study, randomized controlled trial (RCT), non-RCT, quantitative descriptive study, and mixed methods). study, randomized controlled trial (RCT), non-RCT, qu…”
Glarcher, M., & Vaismoradi, M. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106776 · Cited by 26
Ekpor, E., BROBBEY, S. S., Kumah, C. Y., & Akyirem, S. (2025) · International Health · doi:10.1093/inthealth/ihaf060 · Cited by 36
“17 The MMAT was used because it is the recommended assessment tool for mixed methods reviews.”
Sathiyaseelan, G., Hashim, S. M., & Nawi, A. M. (2025) · BMC Public Health · doi:10.1186/s12889-025-23118-z · Cited by 24
“Using the Mixed Methods Appraisal Tool (MMAT), the authors evaluated the quality of each of the 53 studies [25].”
Liu, J., Yu, H., Cheung, W. C., Bleakney, A., & Jan, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326585 · Cited by 13
Tang, X., (曾子欣), Z. Z., Huang, H., & Symonds, J. E. (2025) · ECNU Review of Education · doi:10.1177/20965311251371227 · Cited by 25
“( simple and clear (Hong et al., 2018b), but many existing tools failed to provide it.”
InfluentialPatient-Centered Care for People with Depression and Anxiety: An Integrative Review Protocol
de Pinho, L. G., Correia, T., Lopes, M. J., da Fonseca, C. J. V., Marques, M. D. C., Sampaio, F., & do Arco, H. M. D. S. L. R. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11050411 · Cited by 17
“Quality Appraisal Since this is a review that intends to integrate quantitative, qualitative, and mixed methods studies, the “mixed methods assessment tool” will be used to assess the quality of the selected studies [18].”
MethodCheng, H., & Gan, T. (2022) · Palliative Medicine · doi:10.1177/02692163211058598 · Cited by 18
Tovar, M. A., Pilkington, R. A., Goodwin, T. L., & Root, J. M. (2022) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x22000747 · Cited by 16
Irbık, D. E. (2022) · Frontiers in Water · doi:10.3389/frwa.2022.921459 · Cited by 15
“Quality assessments of all 67 studies were done using the quality appraisal tool MMAT version 2018 a priori for internal and external validity for all eligible studies (Hong et al., 2018).”
MethodMan, H., Wood, L., & Glover, N. (2022) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2780 · Cited by 14
“Evidence was assessed using a checklist approach by the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodA Systematic Review of Therapeutic Process Factors in Mental Health Treatment for Autistic Youth
Albaum, C. S., Vashi, N., Bohr, Y. M., & Weiss, J. A. (2022) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-022-00409-0 · Cited by 16
Mapping research findings on change implementation in nursing practice: A scoping literature review
Salma, I., & Waelli, M. (2022) · Nursing Open · doi:10.1002/nop2.1369 · Cited by 17
Benefits of supportive strategies for carers of people with high-grade glioma: a systematic review
Jones, D., Pinkham, M. B., Wallen, M. P., Hart, N. H., Joseph, R., Strodl, E. S., Ownsworth, T., Beesley, V. L., Crichton, M., & Chan, R. J. (2022) · Supportive Care in Cancer · doi:10.1007/s00520-022-07419-2 · Cited by 19
Cummergen, K., Hannah, L. A., Jopling, L., Cameron, R. A., Walsh, C. M., & Pérez, J. (2022) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2022.100431 · Cited by 15
Kristensen, J. H., Trifunovic, S., Strand, J., Vistnes, K. K., Syvertsen, A., Zandi, A., & Pallesen, S. (2022) · International Gambling Studies · doi:10.1080/14459795.2022.2143856 · Cited by 16
“The Mixed Methods Appraisal Tool 2018 version (MMAT; Hong et al., 2018) was used to assess the risk of bias in the included studies.”
MethodOuellet, S., Gallani, M. C. B. J., Gélinas, C., Fontaine, G., Archambault, P. M., Mercier, É., Severino, F., & Berube, M. (2022) · Nursing Open · doi:10.1002/nop2.1550 · Cited by 21
“3.3.4 | Confidence in cumulative evidence The quality of the evidence will be critically appraised by two independent reviewers (SO, FS) with the Mixed Methods Appraisal Tool (MMAT) v.2018 (Hong et al., 2018; Hong et al., 2019).”
BackgroundMethodInfluentialFeasibility and Cost of Telehealth Head and Neck Cancer Survivorship Care: A Systematic Review
Ravin, E. D., Armache, M., Campbell, F. M., Rising, K. L., Worster, B. K., Handley, N. R., Fundakowski, C., Cognetti, D. M., & Mady, L. J. (2023) · Otolaryngology · doi:10.1002/ohn.213 · Cited by 18
Neubauer, L., Straw, I., Mariconti, E., & Tanczer, L. M. (2023) · Journal of Family Violence · doi:10.1007/s10896-023-00517-7 · Cited by 20
“This paper builds on existing frameworks for assessing ML and mixed methods research (Dreisbach et al., 2019; Hinds et al., 2021; Hong et al., 2018; Siebert et al., 2020) to develop a checklist of 21 ‘yes/no’ criteria which were used to assess the overall quality, reliability and potential bias of…”
MethodTippens, J. A., Erwin, S., Eller, K., Gross, R. M. D., Bearss, B., Kemp, B. R., Mollard, E. K., Njiru, L., Okwarah, P., Palmer-Wackerly, A. L., et al. (2023) · Journal of Immigrant and Minority Health · doi:10.1007/s10903-023-01516-y · Cited by 22
“To appraise the latter, reviewers evaluated each study using the separate MMAT criteria for qualitative or quantitative research [24].”
InfluentialExperiences of nurses and midwives with disabilities: A scoping review
Baker, C., Malik, G., Davis, J., & McKenna, L. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15802 · Cited by 15
Nicolau, V., Brandão, D., Rua, T., & Escoval, A. (2023) · BMC Public Health · doi:10.1186/s12889-023-16485-y · Cited by 16
“Procedures were supported by the validated Mixed Method Appraisal Tool ( MMAT) consisting of 19 core criteria in a quality scoring system grouped into five methodological categories [30–33].”
MethodA scoping review on interpersonal coping in sports
Eckardt, V. C., & Tamminen, K. A. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2023.2251137 · Cited by 17
“The form re fl ects a modi fi ed version of the Mixed Methods Appraisal tool (MMAT; Hong et al., 2018) which accounts for speci fi c characteristics applicable to qualitative, quantitative, and mixed-method studies included in a review.”
Alsolami, F. J., & Tayyib, N. (2023) · International Journal of Urological Nursing · doi:10.1111/ijun.12380 · Cited by 18
Goh, P. R. Q., Ng, G. Y. J., Shorey, S., & Lim, S. (2023) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2023.101464 · Cited by 16
Yang, J., Lim, K. H., Lim, K. T., et al. (2023) · Therapeutic Advances in Chronic Disease · doi:10.1177/20406223231204727 · Cited by 13
Gergov, V., Prevendar, T., Vousoura, E., Ulberg, R., Dahl, H. J., Feller, C. M., Jacobsen, C. F., Karain, A., Milic, B., Poznyak, E., et al. (2023) · Adolescent Research Review · doi:10.1007/s40894-023-00228-6 · Cited by 19
Horgan, S., Hegarty, J. M., Drennan, J., Keane, D., & Saab, M. M. (2023) · Journal of Tissue Viability · doi:10.1016/j.jtv.2023.11.004 · Cited by 16
Paterson, C. C., Roberts, C., Blackburn, J., Joseph, J., Northam, H. L., Wallis, E., Hind, A., Caulfield, R., Barratt, M., Toohey, K. L., et al. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15968 · Cited by 15
Aloweni, F. B. A. B., Gunasegaran, N., Lim, S. H., Leow, B. W. X., Agus, N., Goh, I. H. Q., & Ang, S. Y. (2023) · Journal of Tissue Viability · doi:10.1016/j.jtv.2023.11.007 · Cited by 15
Brown, C., Luzmore, R., O’Donovan, R., Ji, G. X., & Patnaik, S. (2024) · International Journal of Educational Management · doi:10.1108/ijem-09-2023-0447 · Cited by 33
Factors influencing HPV vaccine acceptance among females in mainland China: A systematic review
Dapari, R., Li, M., Chen, X. J., et al. (2024) · Clinical Epidemiology and Global Health · doi:10.1016/j.cegh.2024.101514 · Cited by 13
eHealth literacy in prostate cancer: A systematic review
Jackson, S. R., Yu, P. T., Armany, D., Occhipinti, S., Chambers, S. K., Leslie, S., & Patel, M. I. (2024) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108193 · Cited by 15
Pereira, T. L., Rajendran, P., Nantsupawat, A., & Shorey, S. (2024) · Midwifery · doi:10.1016/j.midw.2024.103956 · Cited by 24
Evans, R. W., Maguet, Z. P., Stratford, G. M., Biggs, A. M., Goates, M. C., Novilla, M. L. B., Frost, M., & Barnes, M. (2024) · Maternal and Child Health Journal · doi:10.1007/s10995-024-03898-3 · Cited by 16
“The majority of articles (n = 27) had the highest MMAT score of 5, indicating high empirical research quality.”
BackgroundInfluentialRiera-Negre, L., Hidalgo-Andrade, P., Ramón, M. R. R., & Gelabert, S. V. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1328247 · Cited by 21
Ricoy-Cano, A. J., Zambrano, C. V., Robles, Y. D. L. F., & Vásquez-Peña, G. E. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241234342 · Cited by 19
“An adapted version of the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018a) was used for the assessment of the studies included.”
InfluentialSoori, B. I. P., Regmi, K., & Pappas, Y. (2024) · Community Mental Health Journal · doi:10.1007/s10597-024-01263-w · Cited by 20
“To facilitate the critical appraisal of the articles included in this SR, the quality assessment tool of studies, known as the Mixed Methods Appraisal Tool version 2018 (MMAT V 2018) (Hong et al., 2018) was used.”
BackgroundMethodBarriers to inclusion of Muslim migrant students in western schools. A systematic review
Torres-Zaragoza, L., & Llorent-Bedmar, V. (2024) · International Journal of Educational Research · doi:10.1016/j.ijer.2024.102363 · Cited by 29
Palliative Care in Early Dementia
de Sola-Smith, K., Gilissen, J., van der Steen, J. T., Mayan, I., den Block, L. V., Ritchie, C. S., & Hunt, L. J. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.05.028 · Cited by 24
Patient participation in surgical wound care in acute care settings: An integrative review
Liosatos, K., Tobiano, G., & Gillespie, B. M. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104839 · Cited by 23
“The quality of individual studies was evaluated, as seen in Appendix B, using the Mixed Methods Appraisal Tool (MMAT) version 2018, which allows quality appraisal of different study designs (Hong et al., 2018a, 2018b).”
MethodThe Teaching Personal and Social Responsibility Model within the School Context: A Systematic Review
Jiménez-Parra, J. F., González‐Víllora, S., & Valenzuela, A. V. (2024) · Quest · doi:10.1080/00336297.2024.2375601 · Cited by 13
“The quality assessment of the review process was carried out following the procedure of previous PM studies (Fernandez-Rio & Iglesias, 2022): (a) initially, the review protocol was registered with the International Platform of Registered Systematic Review and Meta-Analysis Protocols (INPLASY) with the number 202,240,0…”
InfluentialFactors that influence the mental health of bus drivers: A systematic review
Akinlotan, O. O., & Drewry, H. (2024) · Mental Health & Prevention · doi:10.1016/j.mhp.2024.200355 · Cited by 13
Pereira, T. L., Wichaikhum, O., Nantsupawat, A., Rajendrana, P., Baladram, S., & Shorey, S. (2024) · International Journal of Mental Health Nursing · doi:10.1111/inm.13417 · Cited by 22
Antonopoulou, V., Meyer, C., Chadwick, P., Gibson, B., Sniehotta, F. F., Vlaev, I., Vassova, A., Goffe, L., Lorencatto, F., McKinlay, A. R., et al. (2024) · Health Research Policy and Systems · doi:10.1186/s12961-024-01209-4 · Cited by 21
“One author (B.G.) individually conducted a quality appraisal of the included studies using the Mixed Methods Appraisal Tool (MMAT) [57, 58] and the Joanna Briggs Institute Critical Appraisal Checklist (JBI) [59].”
MethodResilience and strengths in the Black autism community in the United States: A scoping review
Davis, A., Telfer, N. A., Artis, J., Abubakare, O., Keller‐Bell, Y., Caruthers, C. E., Jones, D. R., & Pierce, N. P. (2024) · Autism Research · doi:10.1002/aur.3243 · Cited by 20
“” We followed the guidelines of Hong et al. (2018) and did not score the criteria rather we used the criteria to guide our understanding of the strengths and limitations of the quality of these research studies across studies.”
Persuasive strategies in digital interventions to combat internet addiction: A systematic review
Theopilus, Y., Mahmud, A. A., Davis, H., & Octavia, J. R. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105725 · Cited by 18
El-Hagrasy, A. M. A., Karrout, R. H., Mcguinness, A. L., et al. (2025) · BMJ Public Health · doi:10.1136/bmjph-2024-001046 · Cited by 13
Domains of wheelchair users' socio-emotional experiences: Design insights from a scoping review.
Rasoulivalajoozi, M., Cucuzzella, C., & Farhoudi, M. (2025) · Disability and Health Journal · doi:10.1016/j.dhjo.2025.101829 · Cited by 11
Virtual Articulator Software: Accuracy, Usability, and Clinical Applicability: A Systematic Review
Saini, R. S., Alshoail, H. H., Kanji, M. A., Vaddamanu, S. K., Mosaddad, S. A., & Heboyan, A. G. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.03.005 · Cited by 23
“28 Articles were included in the systematic review using qualitative analysis.”
Baker, T., Mundell, N. L., Koorts, H., Pebole, M. M., Rosenbaum, S., Ganakas, E., & Teychenne, M. J. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01735-6 · Cited by 16
“Study quality was assessed independently by two authors (TB and EG) using the Mixed Methods Appraisal Tool (MMAT) [59].”
Landi, G., Andreozzi, M., Pakenham, K. I., Grandi, S., & Tossani, E. (2020) · Diabetic Medicine · doi:10.1111/dme.14271 · Cited by 13
Sin, C. M.-H., Huynh, C., Dahmash, D. T., & Maidment, I. (2021) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2020-002520 · Cited by 16
Scoping Review: Suicide Specific Intervention Programmes for People Experiencing Homelessness
Murray, R. M., Conroy, É., Connolly, M., Stokes, D., Frazer, K., & Kroll, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18136729 · Cited by 16
“Working with individuals who are homeless and are at risk of suicide is an anxiety provoking experience, and in order to support frontline workers, or mental health professionals in implementing prevention strategies or interventions, training is required [35].”
BackgroundMethodInfluentialChen, X., Frennert, S., & Östlund, B. (2021) · Ageing International · doi:10.1007/s12126-021-09417-x · Cited by 16
Patients’ preferences for antiretroviral therapy service provision: a systematic review
Belay, Y. A., Yitayal, M., Atnafu, A., & Taye, F. A. (2021) · Cost Effectiveness and Resource Allocation · doi:10.1186/s12962-021-00310-7 · Cited by 13
Thalen, M., Volkers, K. M., van Oorsouw, W. M. W. J., & Embregts, P. J. C. M. (2021) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12953 · Cited by 13
“As a result, two studies with lower-thanacceptable quality were eliminated from further analyses, in accordance with the MMAT guidelines.”
MethodInfluentialNavas, J. M. M., Taubner, S., Mota, C. P., Henriques, M. I. R. S., Saliba, A., Heinonen, E. O., Ramos, S., Moreno-Peral, P., Volkert, J., Adler, A., et al. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.730921 · Cited by 13
“The Mixed Methods Appraisal Tool (55) was used for the risk of bias and the methodological quality assessment of randomised controlled trials.”
MethodPenn, S., & Brown, N. (2022) · Ubiquitous Learning An International Journal · doi:10.18848/1835-9795/cgp/v15i02/1-18 · Cited by 20
“The MMAT allowed critical appraisal of the different types of study design (qualitative, quantitative, and mixed methods) included in the review; this provided an indication of quality rather than an overall score for comparison.”
BackgroundMethodInfluentialThe impact of implementing patient‐reported measures in routine maternity care: a systematic review
Chen, A., Väyrynen, K., Schmidt, A., Leskelä, R., Torkki, P. M., Heinonen, S. T., Tekay, A., & Acharya, G. (2022) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.14446 · Cited by 13
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.