The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,901–2,000 of 4,578
Health inequities in medical crowdfunding: a systematic review
Cai, Y., Kamarudin, S., Jiang, X., & Zhou, B. (2025) · International Journal for Equity in Health · doi:10.1186/s12939-025-02543-x · Cited by 12
“For this purpose, the Mixed Methods Appraisal Tool (MMAT) by Hong, et al. [23] was employed.”
Fisher, S., Blau, A., & Gendler, Y. (2025) · International Nursing Review · doi:10.1111/inr.70049 · Cited by 12
The Impact of Artificial Intelligence on Elementary School Students’ Learning: A Meta-Analysis
Arkoumanis, G., Sofos, A., Ventista, O. M., Ventistas, G., & Tsani, P. (2025) · Computers in the Schools · doi:10.1080/07380569.2025.2520787 · Cited by 13
Cho, H., Cho, E., You, S. B., Sefcik, J. S., Hodgson, N. A., & Demiris, G. (2025) · Journal of Applied Gerontology · doi:10.1177/07334648251348703 · Cited by 12
“Three authors (HC, EC, SBY) independently assessed the quality of 22 articles using the Mixed Methods Appraisal Tool (Hong et al., 2018).”
Ruan, Y., & Cheung, M. (2025) · Journal of Nursing Scholarship · doi:10.1111/jnu.70035 · Cited by 9
Castanho, L., Martinho, D. V., Saial, A. C., de Gouveia, B. R. F. O., Gouveia, É. R., & Ribeiro, F. (2025) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-025-09730-0 · Cited by 12
“This tool is useful for quali - tative and quantitative sensitivity analyses (Hong et al., 2018).”
InfluentialNizamuddin, M. (2025) · International Journal of Information Security · doi:10.1007/s10207-025-01095-z · Cited by 14
“MMAT is well-suited for mixed research designs and systematically evaluates study objectives, methodological alignment, data collection rigor, and the validity of findings [35, 36].”
O’Keefe, B., Yuen, E. Y. N., Perlen, S., et al. (2025) · Australasian Journal on Ageing · doi:10.1111/ajag.70070 · Cited by 6
Ho, R. C. Y., Wong, S. N., Chia, P., Tang, C., & Ng, M. T. T. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70008 · Cited by 14
“The MMAT version 2018 (Hong et al., 2018) was used for quality assessment by four independent reviewers based on the quantitative, qualitative, and mixed-method studies on this topic.”
Anthonj, C., Setty, K., Höser, C., Brown, J., Ihnacik, L., & Poague, K. I. H. M. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.118526 · Cited by 11
A systematic review of paper‐based and digital board games for collaborative science learning
Othman, M. K., Mat, R., & Sim, K. C. (2025) · Review of Education · doi:10.1002/rev3.70107 · Cited by 6
Jinqiu, L., Tan, J.-Y. B., Wang, H., Wang, T., Li, M., Terry, D. R., Taylor, M., Huang, Y., & Sharma, S. (2025) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2025.102976 · Cited by 6
Factors Influencing Exclusive Breastfeeding During the Postpartum Period: A Mixed-Methods Study
Minarini, G. P. S. S., Lima, E. D. F. A., Figueiredo, K. C., et al. (2025) · Nutrients · doi:10.3390/nu17182992 · Cited by 6
“In addition, the Mixed Methods Appraisal Tool (MMAT) was used to ensure the appropriateness of the study report [13].”
Taylor-Phillips, F., Forshall, G., Jones, G. L., & Turner‐Moore, R. (2025) · Human Fertility · doi:10.1080/14647273.2025.2553529 · Cited by 7
“The MMAT was selected as it can be applied to heterogeneous study designs, incorporating different appraisal questions for qualitative, quantitative, and mixed methods studies (Hong et al., 2018).”
InfluentialBunnewell, S., Keating, S., Parsons, J., Hickey, M., & Hillman, S. (2025) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.70023 · Cited by 13
Mobarki, K., Guo, P., Mobarki, M. I., & Efstathiou, N. (2025) · Nursing in Critical Care · doi:10.1111/nicc.70201 · Cited by 9
“The Mixed Methods Appraisal Tool (MMAT) was employed for mixed methods studies [28].”
Choi, S., Kim, M., Park, J.-E., et al. (2025) · BMC Palliative Care · doi:10.1186/s12904-025-01793-4 · Cited by 6
“As the included studies had various designs (quantitative, qualitative, and mixed methods), the Mixed Meth-ods Appraisal Tool (MMAT) 2018 version was used for quality appraisal [19].”
Parental Factors Associated with Social Media Use in Adolescence: A Systematic Review
Morawska, A., Adina, J., Khan, A., & Turner, K. (2025) · Journal of Adolescence · doi:10.1002/jad.70062 · Cited by 18
Public Health Impacts of Water Fluorides: Current Evidence from a Rapid Systematic Review
Zohoori, F. V., Kumah, E. A., Kronic, J., Drinnan, M. J., & Morris, A. J. (2025) · Advances in Nutrition · doi:10.1016/j.advnut.2025.100547 · Cited by 11
“The quality assessment of all included studies was conducted using the Mixed Methods Appraisal Tool (MMAT) [17].”
A systematic review of science outreach: characteristics and research quality
Guarrella, C., Hill, J. L., Millar, V., Toscano, M., DRIEL, J. H. V., & Chakraborty, D. (2025) · Studies in Science Education · doi:10.1080/03057267.2025.2576976 · Cited by 13
Prasetya, F., Fortuna, A., Samala, A. D., Andriani, W., Bagus, M. I., Renaldi, I., Rawas, S., Criollo-C, S., & Ayasrah, F. T. (2025) · SN Computer Science · doi:10.1007/s42979-025-04488-y · Cited by 10
“The Mixed Methods Appraisal Tool (MMAT) [24, 25] was used to evaluate studies employing mixed methods, which integrated quantitative and qualitative data.”
Teacher leadership in the Arab states: A systematic review
Ghamrawi, N., Shal, T., & Ghamrawi, N. A. R. (2025) · Educational Management Administration & Leadership · doi:10.1177/17411432251406019 · Cited by 9
How Do Older People Experience Person-Centred Integrated Care? An Integrative Review of the Evidence
Murphy, S., McCance, T., & White, P. J. (2025) · International Journal of Integrated Care · doi:10.5334/ijic.9066 · Cited by 10
Communication Strategies to Promote Patient Engagement in Telemedicine: Systematic Review
Hu, Y. M., Ngai, C. S. B., & Jiang, R. (2025) · Journal of Medical Internet Research · doi:10.2196/85456 · Cited by 15
“This tool provides a flexible framework for appraising qualitative, quantitative, and mixed methods studies included in a systematic review [87].”
Rethinking public participation in South Africa’s local governance system: A systematic review
Ndlovu, J., & Muringa, T. P. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2604320 · Cited by 6
Xu, Q., Xu, Y., Liu, X., & Ma, X. (2026) · Journal of Medical Internet Research · doi:10.2196/76880 · Cited by 11
“The study quality assessment was conducted using the 2018 version of the Mixed Methods Appraisal Tool, a tool specifically designed to evaluate the quality of qualitative, quantitative, and mixed methods studies [19].”
Rethinking the digital divide in health: a critical interpretive synthesis of research literature
Bradway, M., Wang, B., Nybakke, H. L., Ingebrigtsen, S. A., Dyb, K., & Rødseth, E. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1683565 · Cited by 10
Personalized Learning With Artificial Intelligence in Dental Education: An Integrative Review
MithunPai, B. H., Yellapurkar, S., Chengappa, K., & Pentapati, K. C. (2026) · International Journal of Dentistry · doi:10.1155/ijod/8928166 · Cited by 7
“…Briggs Institute Critical Appraisal Checklist for Cross Sectional Studies [24] was used, for mixed methods study, the Mixed Methods Appraisal Tool (MMAT), Version 2018 [25] was applied and for narrative review, the Scale for the Assessment of Narrative Review Articles (SANRA) [26] was applied.”
Romano, S., Demaria, F., & Carbone, A. (2026) · Discover Sustainability · doi:10.1007/s43621-026-02623-0 · Cited by 10
“In some cases, certification requirements may inadvertently limit yield potential, particularly when shade tree provisions compete with cocoa plants for resources [67] or in the case of Organic standard, restricting the use of synthetic fertilizers and pesticides [52].”
Siegel, Z., Quinkert, E. M., Pai, J., Miller, C., & Lewis, M. W. (2026) · JMIR Aging · doi:10.2196/70012 · Cited by 7
“The quality assessments were achieved through critical appraisal tools used in Joanna Briggs Institute Systematic Reviews [59] and the McGill Mixed Methods Appraisal Tool Version 2018 [60].”
Implementation of AI in career counselling for university students: a systematic review
Sarmurzin, Y., Azhenov, A., Kudysheva, A., Ualikhanova, B. S., Saparbekova, I., & Besbayev, A. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1787689 · Cited by 15
Edwards, J., Armstrong, M., Begum, S., Dunphy, E., Lau, R., Naughton, M., Taylor, S. J., & Ross, J. (2026) · The Lancet Primary Care · doi:10.1016/j.lanprc.2026.100175 · Cited by 11
Ravalihasy, A., Kardaś-Słoma, L., Yazdanpanah, Y., & Ridde, V. (2022) · Systematic Reviews · doi:10.1186/s13643-022-01970-z · Cited by 6
“This tool allows to assess the design quality of quantitative studies [12, 20, 21].”
MethodKalhor, M., Mohammadi, E., Shahali, S., Amini, L., & Moghaddam-Banaem, L. (2022) · International Journal of Reproductive BioMedicine (IJRM) · doi:10.18502/ijrm.v20i9.12062 · Cited by 14
Geiger, I., Schang, L., & Sundmacher, L. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09461-0 · Cited by 8
“However, as these tools do not include methodological studies in their targeted study designs, their application was not suitable for the purpose of the methodological review [58, 59].”
BackgroundSalgueiro‐Oliveira, A., Rêgo, A. D. S., Santos‐Costa, P., Bernardes, R. A., Filipe, L., Sousa, L. B., Barboza, R., Carvalho, M. A. F., Bouçanova, M., Lopes, M. C. F. D. G., et al. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20186773 · Cited by 8
“This study adopts a mixed research approach, employing the strategy of concomitant data triangulation based on the constructivist approach and following the guidelines of the Mixed Method Appraisal Tool (MMAT) guide [25].”
AlRashed, H., Miele, J., Prasad, J., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0296243 · Cited by 6
Hughes, M., Hanna, K. L., Wiles, A., Taylor, E., & Giebel, C. M. (2024) · Dementia · doi:10.1177/14713012231225797 · Cited by 8
“One paper used mixed methodology and so the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) was used to critically appraise it.”
MethodSecondary school teachers’ conceptions of assessment: A systematic review
Halim, H. A., Hamzah, M. I., & Zulkifli, H. (2024) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102311 · Cited by 19
Dennis, D., Henderson‐Wilson, C., Watson, J., & Lawson, J. T. (2024) · Sustainability · doi:10.3390/su16031077 · Cited by 12
Initial Construction of the Exercise Maintenance Motivation Scale: A Mixed-Method Study
Nam, T. T., Dadras, O., Le, C. N., Phu, D. H., Tin, P. T., An, T. D., Dat, N. T., Du, T. N., Hiền, N. T. T., Châu, T. T. T., et al. (2023) · Journal of Human Earth and Future · doi:10.28991/hef-2023-04-04-06 · Cited by 7
“To ensure the comprehensiveness of this mixed-method design study, checklists follow the guidelines of the Mixed Methods Appraisal Tool (MMAT), version 2018, and the CASP checklist for appraisal of qualitative research, version 2018, designed for information professionals and researchers [46, 47].”
MethodLee, J., Chepkorir, J., Alharthi, A., Adeleye, K. K., & Warren, N. (2024) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-024-01945-x · Cited by 7
“For mixed-methods studies, the Mixed Meth-ods Appraisal Tool [31] was used.”
Equine-assisted services for people living with dementia: a systematic review
Sebalj, M., Lakhani, A., Grindrod, A., & Stuckey, R. (2024) · Alzheimer s Research & Therapy · doi:10.1186/s13195-024-01453-4 · Cited by 10
“Methodological quality of included studies was assessed using the Mixed Methods Appraisal Tool (MMAT), version 2018 [25].”
MethodA model fidelity check in cooperative learning research in physical education
Gallego, D. I., & Fernández‐Río, J. (2024) · Physical Education and Sport Pedagogy · doi:10.1080/17408989.2024.2342831 · Cited by 6
“Two reviewers used the Mixed Methods Assessment Tool (MMAT; Hong et al., 2018) to assess the quality of each study.”
Leadership conceptions of nurses and physicians in emergency care: A scoping review
Rixon, A., Elder, E. G., Bull, C. F., Crilly, J. L., Østervang, C., Frieslich, H., Robertson, S., Pink, E., & Wilson, S. G. (2024) · International Emergency Nursing · doi:10.1016/j.ienj.2024.101454 · Cited by 8
Parent Perspectives of Early Childhood Special Education Transition
Gaspar, C. R., & Jahromi, L. B. (2024) · Infants & Young Children · doi:10.1097/iyc.0000000000000268 · Cited by 7
“The added subitems were consistent across quantitative, qualitative, and mixed MMAT checklists.”
MethodInfluentialChen, Q. (2024) · Current Problems in Surgery · doi:10.1016/j.cpsurg.2024.101542 · Cited by 6
Inayat, S., Aziz, F., Younas, A., & Durante, Á. (2024) · Heart Lung and Circulation · doi:10.1016/j.hlc.2024.05.010 · Cited by 7
Chua, T. X. J., López, V., Chua, Q. W. C., & Lau, S. T. (2024) · Nurse Education Today · doi:10.1016/j.nedt.2024.106380 · Cited by 10
The Impact of Monoclonal Antibody Usage on Hearing Outcomes: A Systematic Review
Arya, P., Salmerón, Y., Quimby, A. E., et al. (2024) · The Laryngoscope · doi:10.1002/lary.31763 · Cited by 6
Sirparanta, A., Touati, C. D., Cyr, C. P., & Miljkovitch, R. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241282995 · Cited by 19
Dubbeldeman, E. M., Jong, J. C. K., Ardesch, F. H., Boelens, M., van der Velde, L. A., van der Steen, S. G. L., Heijnders, M. L., & Crone, M. R. (2024) · Health & Social Care in the Community · doi:10.1155/2024/5597259 · Cited by 8
Pahwa, M., Cernat, A., Abelson, J., Demers, P. A., Schwartz, L., Shen, K., Chowdhury, M., Higgins, C., & Vanstone, M. G. (2024) · Journal of Health Services Research & Policy · doi:10.1177/13558196241288984 · Cited by 9
“39 The MMAT considers key quality domains associated with each of the fi ve study designs included in this review.”
Yan, W. C., Ohlsen, S., & Wood, E. (2024) · British Journal of Occupational Therapy · doi:10.1177/03080226241288157 · Cited by 6
“The Mixed-Methods Appraisal Tool (MMAT) tool version 2018 (Hong et al., 2018) was used to appraise mixed-method studies.”
The use of Language ENvironment Analysis in autism research: A systematic review
Putnam, O. C., Markfeld, J. E., Wright, S. T., Feldman, J. I., Goldblum, J., Karpinsky, M., Neal, A. J., Swanson, M. R., & Harrop, C. (2024) · Autism · doi:10.1177/13623613241290072 · Cited by 12
“Study quality was assessed for full-text articles (e.g. not conference posters) using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Promoting International Students’ Mental Health Unmet Needs: An Integrative Review
Shek, C. H., Chan, S. W., Stubbs, M. A., & Lee, R. (2024) · International Journal of Mental Health Promotion · doi:10.32604/ijmhp.2024.055706 · Cited by 19
The use of virtual patient simulations in psychology: A scoping review
Hossain, S. I., Kelson, J. N., & Morrison, B. W. (2024) · Australasian Journal of Educational Technology · doi:10.14742/ajet.9559 · Cited by 10
Health assets among refugees in Australia: a systematic review
Dawson, A. J., Adjei-Mensah, E., Hayen, A. D., Nathan, S., Heywood, A. E., Mahimbo, A., Merrington, H., & Rogers, C. (2025) · BMC Public Health · doi:10.1186/s12889-024-20915-w · Cited by 10
“…to assess the quality of evidence in the included studies were the CASP Qualitative Studies Checklist [25] for qualitative studies, the Mixed Methods tool [26] for mixed methods studies, the AXIS Tool for cross-sectional studies [27] and the Non-Randomised – Ottawa [27] for the longitudinal study.”
The process of pain assessment in people with dementia living in nursing homes: a scoping review
Overen, C. K., Larsson, M., Hillestad, A. H., Karlsson, I., & Eriksen, S. (2025) · Palliative Care and Social Practice · doi:10.1177/26323524241308589 · Cited by 10
“31 SE and CKO independently assessed 10 studies, and CKO solely assessed the remaining.”
Pope, N., Birnie, K. A., Noel, M. E., Dol, J., Li, D., MacNeil, M., Zientek, D., Surry, V., & Stinson, J. (2025) · Children · doi:10.3390/children12010077 · Cited by 10
“All records were independently evaluated for quality using the Mixed Methods Ap-praisal Tool (MMAT-v2018) as it is applicable across various study types, including qualitative, quantitative RCT, quantitative nonrandomized, quantitative descriptive, and mixed-methods research [53,54].”
A Systematic Literature Review of Trauma Systems: An Operations Management Perspective
Wang, Z., Rostami-Tabar, B., Haider, J. J., Naim, M. M., & Haider, J. (2025) · Advances in Rehabilitation Science and Practice · doi:10.1177/27536351241310645 · Cited by 13
“However, owing to the constraints of MMAT in assessing systematic literature reviews, 22 the Critical Appraisal Skills Programme (CASP) checklists for systematic review and meta-analysis checklist 26 were utilised for the assessment of 6 identified systematic review articles in this review (Table 3…”
Bollinger, A., Fortes, M. S., zu Schwabedissen, H. E. M., Hersberger, K. E., Stäuble, C. K., & Allemann, S. S. (2025) · Pharmacy · doi:10.3390/pharmacy13010014 · Cited by 7
“The qualitative and quantitative components were each evaluated according to the MMAT mixed-methods criteria [19].”
Henry, G., Benson, O. G., Kazemi, F., et al. (2025) · Digital Society · doi:10.1007/s44206-025-00168-9 · Cited by 6
“The Critical Appraisal Skills Programme was used to appraise the quality of the qualitative studies, while the Mixed Methods Appraisal Tool was used for the quantitative and mixed-methods studies (Hong et al., 2018).”
Khan, M. N., Tarpey‐Brown, G., & Block, K. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251325187 · Cited by 10
“We utilized a Mixed Methods Appraisal Tool (version 2018) for qualitative, mixed methods, and quantitative studies to evaluate the quality of the included studies (Hong et al., 2018).”
Dharmayani, P. N. A., Todd, A., Dam, J., Chau, J. Y., Wardak, S., Champion, K. E., Mihrshahi, S., & Partridge, S. R. (2025) · Health Literacy and Communication Open · doi:10.1080/28355245.2025.2489394 · Cited by 6
Ανδρέου, Γ., & Argatzopoulou, A. (2025) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-025-00512-2 · Cited by 13
Rana, T., Chan, D. N. S., Law, B. M. H., Choi, K. C., Shrestha, S., & So, W. K. W. (2025) · PLoS ONE · doi:10.1371/journal.pone.0321627 · Cited by 14
Josefsson, K. A., & Lindroth, M. (2022) · Sex Education · doi:10.1080/14681811.2021.2011188 · Cited by 6
Behrendt, D., Schirmer, M., Wendschuh, V., & Sumngern, C. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055692 · Cited by 12
van Staden, D. B., Chetty, V., & Munsamy, A. J. (2022) · Systematic Reviews · doi:10.1186/s13643-022-01906-7 · Cited by 6
“The Mixed Method Appraisal Tool [9] will be utilised for quality appraisal of all studies accepted after full-text review.”
MethodInfluentialCAUSATIVE FACTOR OF DELAY IN MATERNAL REFERRAL - SYSTEMATIC REVIEW
Amalia, S. R., Lestari, P., & Ningrum, A. G. (2022) · Indonesian Midwifery and Health Sciences Journal · doi:10.20473/imhsj.v6i1.2022.1-14 · Cited by 7
“(Hong et al., 2018) Amalia et al. : Causative Factor of Delay… Indonesian Midwifery and Health Sciences Journal, 2022, 6 (1), 1-14 Indonesian Midwifery and Health Sciences Journal problems regarding poor communication and transparency of information about services.”
BackgroundBaker, J. A., Berzins, K. M., Canvin, K., Kendal, S., Branthonne‐Foster, S., Wright, J. M., McDougall, T., Goldson, B., Kellar, I., & Duxbury, J. A. (2022) · Health and Social Care Delivery Research · doi:10.3310/yvkt5692 · Cited by 11
Oyegoke, O. O., Akoniyon, O. P., Ogunsakin, R. E., Ogunlana, M. O., Adeleke, M. A., Maharaj, R., & Okpeku, M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19116776 · Cited by 6
Charifson, M., Vieira, D. L., Shaw, J., Kehoe, S. M., & Quinn, G. P. (2022) · F&S Reviews · doi:10.1016/j.xfnr.2022.07.004 · Cited by 9
Loizou, S., Fowler, D. G., & Hayward, M. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102191 · Cited by 8
Gardiner, P. M., Kinnafick, F., Breen, K. C., Girardi, A., & Hartescu, I. (2022) · Journal of Forensic Psychiatry and Psychology · doi:10.1080/14789949.2022.2111320 · Cited by 7
“Based on an earlier MMAT (Pluye et al., 2011), each paper was scored against items with answers, ‘yes’, ‘no’ or ‘can’t tell’.”
BackgroundMethodInfluentialMeddick-Dyson, S., Boland, J. W., Pearson, M., Greenley, S. L., Gambe, R. G., Budding, J. R., & Murtagh, F. E. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02054-8 · Cited by 7
“The Mixed Methods Appraisal Tool has been assessed for reliability and efficiency and offers a comprehensive way to assess studies of numerous method types and produces an overall score as a percentage [36].”
MethodInfluentialStrategies to Address Gendered Racism in Science Research Careers: a Scoping Review
Salmon, U. (2022) · Journal for STEM Education Research · doi:10.1007/s41979-022-00079-1 · Cited by 9
Alejandro, A. L., Leo, C., & Bruce, M. (2022) · Health Communication · doi:10.1080/10410236.2022.2149132 · Cited by 6
“The Mixed Methods Appraisal Tool (Hong et al., 2018) was used to critically appraise the studies (Hong et al., 2018).”
MethodUnmet Care Needs in Breast Cancer Survivors: An Integrative Review
Tabari-Khomeiran, R., Ahmadi, F., Jafaraghaee, F., Noveiri, M. J. S., & Mirfarhadi, N. (2023) · Journal of Holistic Nursing and Midwifery · doi:10.32598/jhnm.33.1.2272 · Cited by 7
McGowan, D., Mather, C. A., & Stirling, C. (2023) · Nursing Reports · doi:10.3390/nursrep13010020 · Cited by 9
“The Mixed Methods Appraisal Tool (MMAT) [27] was used to appraise the quality of quantitative, qualitative and mixed-methods studies.”
MethodCameranesi, M., Adisaputri, G., Jefferies, P., & Ungar, M. T. (2023) · Health Sciences Review · doi:10.1016/j.hsr.2023.100090 · Cited by 7
Parey, B., & Kutscher, E. L. (2023) · Journal of Mixed Methods Research · doi:10.1177/15586898231175192 · Cited by 7
“Other tools rely on a dichotomous response (e.g., MMAT ’ s “ yes ” or “ no ” ; Hong et al., 2018), which may not capture the nuances of applying mixed methods approaches.”
MethodWhat Is Important to the Younger Person (≤50 Years) When Having a Total Hip Arthroplasty
Mew, L. E., Heaslip, V., Immins, T., & Wainwright, T. W. (2023) · Orthopaedic Nursing · doi:10.1097/nor.0000000000000955 · Cited by 7
Dewi, E. A. S., Sanofi, Z., Pratamawaty, B. B., & Arifin, H. S. (2023) · Jurnal Komunikasi Malaysian Journal of Communication · doi:10.17576/jkmjc-2023-3903-17 · Cited by 12
Technologies, Physician’s Caring Competency, and Patient Centered Care:A Systematic Review
Bollos, L. A. C., Zhao, Y., Soriano, G. P., Tanioka, T., Otsuka, H., & Locsin, R. C. (2023) · The Journal of Medical Investigation · doi:10.2152/jmi.70.307 · Cited by 8
Layne, D. M., Logan, A., & Lindell, K. O. (2024) · Nursing Reports · doi:10.3390/nursrep14030130 · Cited by 7
“The Mixed Methods Appraisal Tool (MMAT) [40] was used to assess the quality of all relevant articles after full-text review.”
MethodHappell, B., Jacob, A. M., Furness, T., Stimson, A., Curtis, J., Watkins, A., Platania‐Phung, C., Scholz, B., & Stanton, R. (2024) · Journal of Mental Health · doi:10.1080/09638237.2024.2390364 · Cited by 7
“Quality appraisal of the final studies for inclusion was performed independently by two reviewers using the Mixed Methods Appraisal Tool (MMAT, version 2018) (Hong et al., 2018).”
InfluentialHanvey, G. A., Johnson, H., Cartagena, G., Dede, D. E., Krieger, J. L. R., Ross, K. M., & Pereira, D. B. (2024) · Journal of Clinical and Translational Science · doi:10.1017/cts.2024.677 · Cited by 11
El‐Den, S., McMillan, S. S., Wheeler, A., Ng, R., Roennfeldt, H., & O’Reilly, C. L. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038270 · Cited by 6
Arora, A. K., Quesnel‐Vallée, A., Lessard, D., Mate, K., Rodriguez-Cruz, A., Kronfli, N., Engler, K., Vedel, I., & Lebouché, B. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-040646 · Cited by 6
Self-citationLee, H., Kim, E., Morse, B. L., & Lee, S. E. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18136841 · Cited by 6
“The quality of the selected articles was appraised using the mixed-method appraisal tool (MMAT), which evaluates the methodological quality of qualitative, quantitative, and mixed-methods studies [21].”
MethodMobile applications for prematurity: a systematic review protocol
Sadeghi, M., Kahouei, M., Pahlevanynejad, S., Valinejadi, A., Momeni, M., Kermani, F., & Seddighi, H. (2021) · BMJ Paediatrics Open · doi:10.1136/bmjpo-2021-001183 · Cited by 10
Erwin, J., Paisi, M., Witton, R., Neill, S., Burns, L., Vassallo, I., Nelder, A., Facenfield, J., Devalia, U., & Vassallo, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182312346 · Cited by 13
Gilbert, E., Turner, M., de Viggiani, N., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-060886 · Cited by 6
Baker, J. A., Kendal, S., Berzins, K. M., Canvin, K., Branthonne‐Foster, S., McDougall, T., Goldson, B., Kellar, I., Wright, J. M., & Duxbury, J. A. (2022) · Children & Society · doi:10.1111/chso.12581 · Cited by 10
Evaluation of mobile emergency service with the use of mixed method
de Oliveira, C. C. M., dos Santos, J. L. G., & Novaes, H. M. D. (2022) · Cadernos de Saúde Pública · doi:10.1590/0102-311xen096221 · Cited by 6
Nwagha, T. U., Nweke, M., & Ezigbo, E. D. (2022) · Hematology · doi:10.1080/16078454.2022.2107908 · Cited by 8
“The mixed method appraisal tool (MMAT) Version 2011 was used to assess quality [14].”
MethodThe underpinning of meaningful activities by brain correlates: a systematic review
Cruyt, E., VRIENDT, P. D., Geyter, N. D., Leirsberghe, J. V., Santens, P., Baets, S. D., Letter, M. D., Vlerick, P., Calders, P., Pauw, R. D., et al. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1136754 · Cited by 9
“, 2014) and 14 studies did not include personally relevant goals (Raichle, 2000; Walter et al., 2001; Gutman and Schindler, 2007; Caspers et al., 2010; Schweizer et al., 2013; Rickard and Vella-Brodrick, 2014; Chen et al., 2015; Ikiugu et al., 2016; Kong et al., 2016; Orrison et al., 2017; Hong et al., 2018; Cieza et…”
BackgroundMethodKim, Y., Lee, H., Lee, H., Lee, M., Kim, S., & Konlan, K. D. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e20738 · Cited by 9
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.