The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 301–400 of 4,578
A systematic review of social support for siblings of children with neurodevelopmental disorders
Kirchhofer, S. M., Orm, S., Haukeland, Y. B., Fredriksen, T., Wakefield, C. E., & Fjermestad, K. W. (2022) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2022.104234 · Cited by 67
Freedman, B., Li, W. W., Liang, Z., Hartin, P., & Biedermann, N. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16111 · Cited by 71
Mulungu, K., Kassie, M. B., & Tschopp, M. (2025) · Information Technology for Development · doi:10.1080/02681102.2025.2456232 · Cited by 107
Family caregiver's experience of caring for an older adult with delirium: A systematic review
Shrestha, P., & Fick, D. M. (2020) · International Journal of Older People Nursing · doi:10.1111/opn.12321 · Cited by 58
Telehealth Use for Enhancing the Health of Rural Older Adults: A Systematic Mixed Studies Review
Rush, K. L., Singh, S., Seaton, C. L., Burton, L., Li, E., Jones, C., Davis, J. C., Hasan, K., Kern, B., & Janke, R. (2021) · The Gerontologist · doi:10.1093/geront/gnab141 · Cited by 61
A Comprehensive Analysis of Factors Associated with Intimate Partner Femicide: A Systematic Review
Garcia-Vergara, E., Almeda, N., Ríos, B. M., Becerra‐Alonso, D., & Fernández‐Navarro, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19127336 · Cited by 73
“The mentioned guideline is a unique, efficient appraisal tool focused on the methodology criteria of different study designs simultaneously [50].”
MethodDeterminants of Self-Care and Home-Based Management of Hypertension: An Integrative Review
Konlan, K. D., & Shin, J. (2023) · Global Heart · doi:10.5334/gh.1190 · Cited by 91
Micropollutant Removal Efficiency of Advanced Wastewater Treatment Plants: A Systematic Review
Belete, B., Desye, B., Ambelu, A., & Yenew, C. (2023) · Environmental Health Insights · doi:10.1177/11786302231195158 · Cited by 62
Effectiveness and User Experience of Virtual Reality for Social Anxiety Disorder: Systematic Review
Shahid, S., Kelson, J. N., & Saliba, A. J. (2023) · JMIR Mental Health · doi:10.2196/48916 · Cited by 58
“Although the MMAT [46] was used to assess the quality of the studies, there is still a risk of subjective reviewer bias in addressing its criteria.”
MethodInfluentialLouart, S., Bonnet, E., & Ridde, V. (2020) · Health Policy and Planning · doi:10.1093/heapol/czaa093 · Cited by 58
“According to the categories in the Mixed Methods Appraisal Tool (Hong et al. , 2018), the evaluation methods used were descriptive quantitative ( n ¼ 16), mixed methods ( n ¼ 12), randomized quantitative ( n ¼ 7), quantitative without randomization ( n ¼ 11) and qualitative ( n ¼ 14), as shown in Figure 7.”
Homer, C. S., Turkmani, S., Wilson, A. N., Vogel, J. P., Shah, M. G., Fogstad, H., & Langlois, E. V. (2022) · BMJ Global Health · doi:10.1136/bmjgh-2021-006872 · Cited by 48
Brown, M. J., McCann, E., Truesdale, M., Linden, M., & Marsh, L. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17207568 · Cited by 65
“The mixed-methods appraisal tool (MMAT) was used in the assessment of the quality of the included studies [36].”
MethodEvlat, G., Wood, L., & Glover, N. (2021) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2600 · Cited by 56
“Organisational barriers included inadequate support from management and a lack of resources (Berry & Haddock, 2008; Ince et al., 2016).”
BackgroundInfluentialDijkman, B. L., Luttik, M. L., van der Wal‐Huisman, H., Paans, W., & van Leeuwen, B. L. (2021) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2021.11.003 · Cited by 55
Panday, J., Taneja, S., Popoola, A., Pack, R., Greyson, D. L., McDonald, S. D., Black, M., Murray‐Davis, B., Darling, E., & Vanstone, M. G. (2021) · Family Practice · doi:10.1093/fampra/cmab146 · Cited by 52
Supply chain risk management research in construction: a systematic review
Pham, H. T., Pham, T., Quang, H. T., & Dang, C. N. (2022) · International Journal of Construction Management · doi:10.1080/15623599.2022.2029677 · Cited by 73
“Specifically, to implement the MMA method, it is important to firstly respond to the following two questions: (1) ‘Are there clear research questions?’ and (2) ‘Do the collected data allow to address the research questions?’ Responding ‘no’ or ‘cannot tell’ to one or both questions could indicate that the paper is not…”
MethodInfluentialVanstone, M. G., Panday, J., Popoola, A., Taneja, S., Greyson, D. L., McDonald, S. D., Pack, R., Black, M., Murray‐Davis, B., & Darling, E. (2022) · Journal of Midwifery & Women s Health · doi:10.1111/jmwh.13363 · Cited by 51
“Concerning quality appraisal, the MMAT tool discourages the calculation of an overall score from the ratings of each category, but the quality of included papers was mostly acceptable 42 .”
MethodThe role of telehealth startups in healthcare service delivery: A systematic review
Chakraborty, I., Edirippulige, S., & Ilavarasan, P. V. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105048 · Cited by 58
Student expectations of teaching and learning when starting university: a systematic review
Tomlinson, A., Simpson, A., & Killingback, C. (2023) · Journal of Further and Higher Education · doi:10.1080/0309877x.2023.2212242 · Cited by 83
Digital literacy among Korean older adults: A scoping review of quantitative studies
Kang, H., Baek, J., Chu, S. H., & Choi, J. (2023) · Digital Health · doi:10.1177/20552076231197334 · Cited by 72
“We reported overall quality scores using asterisks: 0 (None, indicating none of the quality criteria were met) to 100% (*****, indicating all five criteria were met).(43) This system has been used for quality assessment in descriptive quantitative reviews that examined older adults as the main target population.”
MethodKim, E., Koo, Y., & Nam, I. (2024) · Healthcare · doi:10.3390/healthcare12111090 · Cited by 95
“To assess the risk of bias, we employed the 2018 version of the MMAT checklist [24], which assesses the quality of mixed-methods empirical studies.”
MethodAlgarni, Y. A., Saini, R. S., Vaddamanu, S. K., Quadri, S. A., Gurumurthy, V., Vyas, R., Baba, S. M., Avetisyan, A., Mosaddad, S. A., & Heboyan, A. G. (2024) · Journal of Dental Education · doi:10.1002/jdd.13619 · Cited by 73
Popov, V., Mateju, N., Jeske, C., & Lewis, K. O. (2024) · Annals of Medicine · doi:10.1080/07853890.2024.2424450 · Cited by 83
“To appraise the quality, study design, and assess the risk of bias of included studies, we utilized the Mixed Methods Appraisal Tool (MMAT) [22] (Table 1).”
Johar, H., Abdulsalam, F. I., Guo, Y., Bärnighausen, T., Jahan, N. K., Watterson, J., Leder, K. S., Gouwanda, D., Ramanathan, G. R. L., Lee, K. K. C., et al. (2025) · The Lancet Planetary Health · doi:10.1016/s2542-5196(25)00007-5 · Cited by 79
Wolff, A. C., Dresselhuis, A., Hejazi, S., Dixon, D., Gibson, D., Howard, A. F., Liva, S., Astle, B., Kirkham, S. R., Noonan, V. K., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01725-2 · Cited by 63
Dittborn, M., Turrillas, P., Maddocks, M. M., & Léniz, J. (2021) · Palliative Medicine · doi:10.1177/02692163211029514 · Cited by 57
Sacre, M., Albert, R., & Hoe, J. (2021) · International Journal of Mental Health Nursing · doi:10.1111/inm.12968 · Cited by 57
Ye, B., Chu, C. H., Bayat, S., Babineau, J., How, T.-V., & Mihailidis, A. (2023) · Journal of Medical Internet Research · doi:10.2196/46188 · Cited by 59
Nguyen, L., & Walters, J. (2024) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2024.102336 · Cited by 67
Bonett, S., Li, Q., Sweeney, A., Gaither-Hardy, D., & Safa, H. (2024) · AIDS and Behavior · doi:10.1007/s10461-024-04366-3 · Cited by 49
“Using the Mixed Methods Appraisal Tool [ 24], two authors (AS and SB) independently conducted a critical appraisal of all included studies.”
MethodAlomar, D., Almashmoum, M., Eleftheriou, I., Whelan, P., & Ainsworth, J. D. (2024) · Journal of Medical Internet Research · doi:10.2196/56473 · Cited by 76
“Given the exploratory nature of this review and its limited research evidence and in accordance with the suggestion of Hong et al [22], we opted not to exclude the study meeting only 2 of the MMAT criteria from the final review.”
InfluentialMohsin-Shaikh, S., Furniss, D. J., BLANDFORD, A. E., McLeod, M. C., Ma, T., Beykloo, M. Y., & Franklin, B. D. (2019) · BMC Health Services Research · doi:10.1186/s12913-019-4554-7 · Cited by 44
“The Mixed Methods Appraisal Tool (MMAT) version 2018.”
MethodInfluentialPuszka, S., Walsh, C., Markham, F., Barney, J., Yap, M., & Dreise, T. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.115047 · Cited by 53
“We assessed research quality using the Mixed Methods Appraisal Tool (MMAT), a validated and widely-used tool for assessing research conducted through a wide variety of study designs (Hong et al., 2018; Pace et al., 2012; Pluye et al., 2009).”
MethodDigital Storytelling as a Method in Health Research: A Systematic Review
West, C., Rieger, K. L., Kenny, A. J., Chooniedass, R., Mitchell, K. M., Klippenstein, A. W., Zaborniak, A., Demczuk, L., & Scott, S. D. (2022) · International Journal of Qualitative Methods · doi:10.1177/16094069221111118 · Cited by 76
“The Joanna Briggs Institute Checklist for Qualitative Research (Joanna Briggs Institute, 2017), the Mixed Methods Appraisal Tool (Gagnon et al., 2018; Hong et al., 2018), or the EPHPP Quality Assessment Tool for Quantitative Studies (Thomas et al., 2004) were used for quality appraisal.”
MethodA systematic review of school-based positive psychology interventions to foster teacher wellbeing
Vo, D. T., & Allen, K. (2022) · Teachers and Teaching · doi:10.1080/13540602.2022.2137138 · Cited by 71
Research with and by children: A systematic literature review
Bakhtiar, A., Lang, M., Shelley, B., & West, M. (2023) · Review of Education · doi:10.1002/rev3.3384 · Cited by 65
Mekonnen, A. G., & Mittiku, Y. M. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0000941 · Cited by 74
Nutritional knowledge of nursing students: A systematic literature review
Mancin, S., Sguanci, M. E., Cattani, D., Soekeland, F., Axiak, G., Mazzoleni, B., Marinis, M. G. D., & Piredda, M. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105826 · Cited by 58
Wang, W., Schweickle, M. J., Arnold, E. R., & Vella, S. A. (2025) · Sports Medicine · doi:10.1007/s40279-024-02173-3 · Cited by 89
“The Mixed Methods Appraisal Tool (MMAT; version 2018) was used to appraise the methodological quality of the included studies [45].”
Guillaumie, L., Boiral, O., Baghdadli, A., & Mercille, G. (2020) · Canadian Journal of Public Health · doi:10.17269/s41997-020-00394-3 · Cited by 61
“No study was excluded based on the quality threshold, in accordance with the MMAT (Supplementary Table 2).”
MethodInfluentialLalani, K. H., Crawford, J., & Butler-Henderson, K. A. (2021) · International Journal of Leadership in Education · doi:10.1080/13603124.2021.1988716 · Cited by 58
“, and utilized rigorous use of the PRISMA approach, double screening, double full-text screening, and single quality assessments (Crawford et al., 2020; Hong et al., 2018).”
Nurses’ competency in electrocardiogram interpretation in acute care settings: A systematic review
Chen, Y., Kunst, E. L., Nasrawi, D., Massey, D. L., Johnston, A. N. B., Keller, K. B., & Lin, F. F. (2022) · Journal of Advanced Nursing · doi:10.1111/jan.15147 · Cited by 57
Social Needs Resource Connections: A Systematic Review of Barriers, Facilitators, and Evaluation
Steeves-Reece, A. L., Totten, A. M., Broadwell, K. D., Richardson, D. M., Nicolaidis, C., & Davis, M. M. (2022) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2021.12.002 · Cited by 51
Family presence during adult resuscitation from cardiac arrest: A systematic review
Considine, J., Eastwood, K., Webster, H. K., Smyth, M. A., Nation, K. J., Greif, R. T., Dainty, K. N., Finn, J., & Bray, J. (2022) · Resuscitation · doi:10.1016/j.resuscitation.2022.08.021 · Cited by 45
The impact of nurse leadership education on clinical practice: An integrative review
Page, A., Halcomb, E., & Sim, J. (2021) · Journal of Nursing Management · doi:10.1111/jonm.13393 · Cited by 52
Arora, A. K., Ortiz-Paredes, D., Engler, K., Lessard, D., Mate, K., Rodriguez-Cruz, A., Kronfli, N., Vedel, I., Cox, J., Quesnel‐Vallée, A., et al. (2021) · AIDS Patient Care and STDs · doi:10.1089/apc.2021.0079 · Cited by 52
Self-citationFàbregues, S., Sáinz, M., Romano, M., Escalante-Barrios, E., Younas, A., & López-Pérez, B.-S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.956300 · Cited by 48
Self-citationTaste your way across the globe: a systematic review of gastronomy tourism literature (2000-2021)
Yong, R. Y. M., Chua, B., Han, H., & Kim, B. (2022) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2023.2184445 · Cited by 57
“The Mixed Method Appraisal Tool (MMAT) was utilized to conduct a quality assessment procedure according to the designated standard (Hong et al., 2018).”
MethodBoderie, N. W., Sheikh, A., Lo, E., Sheikh, A., Alex Burdorf, A., van Lenthe, F. J., Mölenberg, F. J. M., & Been, J. V. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.101982 · Cited by 61
Digital play in the early years: A systematic review
Chu, C. P. Y., Paatsch, L. E., Kervin, L. K., & Edwards, S. (2024) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2024.100652 · Cited by 121
Organizational readiness for change: A systematic review of the healthcare literature
Caci, L., Nyantakyi, E., Blum, K., Sonpar, A., Schultes, M.-T., Albers, B., & Clack, L. (2025) · Implementation Research and Practice · doi:10.1177/26334895251334536 · Cited by 94
“The authors of the MMAT discourage computing a numerical overall score of study ratings, as this would not provide information on speci fi c low-quality aspects of the studies (Hong et al., 2018).”
Civic engagement in serious illness, death, and loss: A systematic mixed-methods review
D’Eer, L., Quintiens, B., den Block, L. V., Dury, S., Deliëns, L., Chambaere, K., Smets, T., & Cohen, J. (2022) · Palliative Medicine · doi:10.1177/02692163221077850 · Cited by 62
Seaman, C. E., Green, E., & Freire, K. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19095363 · Cited by 53
Hlongwa, M., Hlongwana, K. W., Makhunga, S., Choko, A. T., Dzinamarira, T., Conserve, D. F., & Tsai, A. C. (2022) · AIDS and Behavior · doi:10.1007/s10461-022-03800-8 · Cited by 44
“All included articles were critically assessed using the Mixed Methods Appraisal Tool (MMAT), version 2018 [41, 42].”
MethodCunha, A. S., Pedro, A. R., & Cordeiro, J. V. (2023) · Journal of Medical Internet Research · doi:10.2196/44188 · Cited by 55
Deka, C., Shrivastava, A., Abraham, A. K., Nautiyal, S., & Chauhan, P. (2025) · Speech, Language and Hearing · doi:10.1080/2050571x.2024.2359274 · Cited by 44
Fu, G., Xu, L., Chen, H., & Lin, J. (2025) · BMC Surgery · doi:10.1186/s12893-025-02763-6 · Cited by 80
Rangachari, P., & Thapa, A. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12494-2 · Cited by 63
“Finally, articles had to meet the critical appraisal criteria set by the JBI [33] and/or the Mixed-Method Assessment Tool (MMAT) [34] as applicable, after satisfying all other eligibility requirements.”
Olsson, A. B. S., Strøm, A., Haaland‐Øverby, M., Fredriksen, K., & Stenberg, U. (2020) · Patient Education and Counseling · doi:10.1016/j.pec.2020.02.028 · Cited by 55
Hong, S., Kim, H., & Park, M. K. (2021) · International Journal of Mental Health Nursing · doi:10.1111/inm.12875 · Cited by 50
Thompson, Z., Baker, F. A., Tamplin, J., & Clark, I. N. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.764372 · Cited by 56
Training and development in sport officials: A systematic review
Cunningham, I., Mergler, J., & Wattie, N. (2022) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.14128 · Cited by 58
Vanclooster, S., Bissell, S., van Eeghen, A. M., Chambers, N. J., Waele, L. D., Byars, A. W., Capal, J. K., Cukier, S., Davis, P. E., Flinn, J., et al. (2022) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-022-09423-3 · Cited by 46
“ADHD: Attention deficit hyperactivity disorder; ADOS‑2: Autism Diagnostic Observation Schedule, Second Edition; AOSI: Autism Observation Scale for Infants; ARRIVE: Animal Research Reporting of In Vivo Experiments; DSM‑5: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition; EEG: Electroencephalography;…”
MethodInfluentialHsueh, Y.-C., Batchelor, R., Liebmann, M., Dhanani, A., Vaughan, L., Fett, A.-K. J., Mann, F., & Pitman, A. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19084766 · Cited by 49
“For this study we judged that 80% of quality criteria in the MMAT scale were met, but had concerns about sampling bias with overrepresentation of females, limiting the generalisability of the findings to men.”
MethodInfluentialNicholson, C. J., Combes, S., Mold, F., King, H., & Green, R. (2022) · Palliative Medicine · doi:10.1177/02692163221118230 · Cited by 50
Makhmud, A., Thornicroft, G., & Gronholm, P. C. (2022) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796022000622 · Cited by 70
Cheah, Y., Wong, H. K., Spitzer, M., & Coutinho, E. (2022) · Music & Science · doi:10.1177/20592043221134392 · Cited by 71
Taylor, L. C., Law, K., Hutchinson, A. M., Dennison, R. A., & Usher‐Smith, J. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0279201 · Cited by 48
Wilson, C., Janes, G., Lawton, R., & Benn, J. (2023) · BMJ Quality & Safety · doi:10.1136/bmjqs-2022-015634 · Cited by 50
Whelan, J., Fraser, P., Bolton, K. A., Love, P., Strugnell, C., Boelsen‐Robinson, T., Blake, M. R., Martin, E., Allender, S. E., & Bell, C. (2023) · Health Research Policy and Systems · doi:10.1186/s12961-023-01023-4 · Cited by 51
“Unlike other quality appraisal tools, the MMAT discourages the use of an overall score (Additional file 3).”
MethodInfluentialYeboah, E. A., Adegboye, A. R. A., & Kneafsey, R. (2023) · The Journal of Climate Change and Health · doi:10.1016/j.joclim.2023.100290 · Cited by 66
“Assessment of methodological quality The quantitative papers and mixed methods papers were appraised using the Mixed Methods Assessment Tool [MMAT] version 2018 [22].”
Bianchi, L. A., Harris, R. G., & Fitzpatrick, J. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.16018 · Cited by 56
“…Management (CEBMa) ‘Critical Appraisal of Survey’ tool for surveys (CEBMa, 2014); the Joanna Briggs Institute Critical Appraisal Tools for observational and prospective designs (JBI, 2020a , 2020b ); and the Mixed-Methods Appraisal Tool (MMAT) for mixed-methods studies (Hong et al., 2018).”
MethodFisher, R., Tran, Q., & Verezub, E. (2024) · Innovation in Language Learning and Teaching · doi:10.1080/17501229.2024.2302984 · Cited by 85
“To assess the quality of methodology applied and reported we used the Mixed Methods Appraisal Tool (MMAT) (Hong et al. 2018).”
InfluentialAnalysing health system capacity and preparedness for climate change
Braithwaite, J., Leask, E., Smith, C. L., Dammery, G., Brooke-Cowden, K., Carrigan, A., McQuillan, E., Ehrenfeld, L., Coiera, E. W., Westbrook, J. I., et al. (2024) · Nature Climate Change · doi:10.1038/s41558-024-01994-4 · Cited by 52
“…and quality of included publications, four quality appraisal tools were applied, depending on publication type: (1) mixed-methods appraisal tool 188 , (2) scale for the assessment of narrative review articles 189 and (3) Joanna Briggs Institute critical appraisal checklist for systematic…”
Hodgson, C. R., Mehra, R., & Franck, L. S. (2024) · Children · doi:10.3390/children11080949 · Cited by 85
“We conducted a quality appraisal using our own adaptation of the mixed-methods appraisal tool (MMAT) [26].”
MethodMcCann, E., Donohue, G., & Brown, M. J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18073403 · Cited by 58
“The qualitative component of the Mixed-Methods Appraisal Tool (MMAT) was used in the assessment of the quality of the included studies [26].”
MethodKonlan, K., Vivor, N. K., Gegefe, I., & Hayford, L. (2022) · BMC Public Health · doi:10.1186/s12889-022-13347-x · Cited by 40
Digital Storytelling as an Intervention for Older Adults: A Scoping Review
Chang, H., Do, Y., & Ahn, J.-Y. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20021344 · Cited by 51
Beheshtaeen, F., Torabizadeh, C., Khaki, S., Abshorshori, N., & Vizeshfar, F. (2023) · Nursing Ethics · doi:10.1177/09697330231221196 · Cited by 52
Boaz, A., Goodenough, B., Hanney, S. R., & Soper, B. (2024) · Health Research Policy and Systems · doi:10.1186/s12961-024-01187-7 · Cited by 55
“Study quality was assessed using the mixed-methods appraisal tool (MMAT v2018), on a scale of 1 (low) to 5 (high) [47].”
MethodCzosnek, L., Richards, J. A., Zopf, E. M., Cormie, P., Rosenbaum, S., & Rankin, N. M. (2021) · BMC Cancer · doi:10.1186/s12885-021-08196-7 · Cited by 55
“The Mixed Methods Appraisal Tool was used to critically appraise studies that described a mixed method design [44].”
MethodScoping review of augmented renal clearance in critically ill pediatric patients
Rhoney, D. H., Metzger, S., & Nelson, N. R. (2021) · Pharmacotherapy The Journal of Human Pharmacology and Drug Therapy · doi:10.1002/phar.2617 · Cited by 42
Lillekroken, D., Halvorsrud, L., Gulestø, R. J. A., & Bjørge, H. (2021) · Journal of Clinical Nursing · doi:10.1111/jocn.16127 · Cited by 47
Enablers and barriers to healthy eating among hospital nurses: A systematic review
Marko, S., Wylie, S., & Utter, J. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104412 · Cited by 52
Protocol for a systematic review of living labs in healthcare
Archibald, M. M., Wiebe, S., Rieger, K. L., Linton, J., & Woodgate, R. L. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-039246 · Cited by 46
Spacey, A., Porter, S., M. Board, & Scammell, J. (2021) · Palliative Medicine · doi:10.1177/02692163211029806 · Cited by 35
Safari, K., McKenna, L., & Davis, J. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104221 · Cited by 44
Collins, J., Horton, K., Ives, E. G.-S., Murphy, G. H., & Barnoux, M. F. L. (2022) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-022-05590-3 · Cited by 81
Shin, J. W., Choi, J., & Tate, J. A. (2022) · Heart & Lung · doi:10.1016/j.hrtlng.2022.12.004 · Cited by 49
Crisis plans in mental health: A scoping review
Cassivi, C., Sergerie‐Richard, S., Saint‐Pierre, B., & Goulet, M. (2023) · International Journal of Mental Health Nursing · doi:10.1111/inm.13158 · Cited by 43
Jones, G. L., Budds, K., Taylor-Phillips, F., Musson, D., Raymer, J., Churchman, D., Kennedy, S., & Jenkinson, C. (2023) · Human Reproduction Update · doi:10.1093/humupd/dmad029 · Cited by 55
“All included studies were assessed for quality using the Mixed Methods Appraisal Tool (MMAT) by K.B. and were discussed during the team meetings if there was any uncertainty (Hong et al. , 2018).”
MethodCanfell, O. J., Woods, L. S., Meshkat, Y., Krivit, J., Gunashanhar, B., Slade, C., Burton‐Jones, A., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/47715 · Cited by 90
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [34].”
Chicote-Beato, M., González‐Víllora, S., Osma, A. R. B., & Navarro, R. (2024) · Aggression and Violent Behavior · doi:10.1016/j.avb.2024.101938 · Cited by 91
Livanis, E., Voultsos, P. P., Vadikolias, Κ., Pantazakos, P., & Tsaroucha, A. K. (2024) · Cureus · doi:10.7759/cureus.58243 · Cited by 62
Inampudi, S., Rajkumar, E., Gopi, A., Vany Mol, K. S., & Sruthi, K. S. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03081-7 · Cited by 53
“The mixed-method studies were appraised with the MMAT (Hong et al., 2018).”
Critically appraising the cass report: methodological flaws and unsupported claims
Noone, C., Southgate, A., Ashman, A., Quinn, É., Comer, D., Shrewsbury, D., Ashley, F., Hartland, J., Paschedag, J., Gilmore, J., et al. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02581-7 · Cited by 88
“This is not recommended by the authors of the MMAT, who actively discourage the use of an overall score of quality [32].”
Educational practices in motion: a scoping review of embodied learning approaches in school
Faella, P., Digennaro, S., & Iannaccone, A. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1568744 · Cited by 108
A systematic review of clinicians’ acceptance and use of clinical decision support systems over time
Newton, N., Bamgboje‐Ayodele, A., Forsyth, R., Tariq, A., & Baysari, M. T. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01662-7 · Cited by 60
“The methodological quality of included studies was independently appraised by two authors (NN and AB, RF, AT or MB) using the well-established MMAT 97 .”
Systematic reviews: A brief historical overview
Hong, Q. N., & Pluye, P. (2018) · Education for Information · doi:10.3233/efi-180219 · Cited by 73
“More specifically, the second paper will present a toolkit for conducting and reporting mixed studies reviews (Pluye et al., 2018) and this third paper will focus on a critical appraisal tool that was developed for use in this type of review (Hong et al., 2018).”
Self-citationRollet, Q., Tron, L., Mil, R. D., Launoy, G. D., & Guillaume, É. (2021) · Preventive Medicine · doi:10.1016/j.ypmed.2021.106692 · Cited by 48
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.