The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,101–1,200 of 4,578
Teleophthalmology and its evolving role in a COVID-19 pandemic: A scoping review
Chong, J. C., Tan, C.-H. N., & Chen, D. Z. (2021) · Annals of the Academy of Medicine Singapore · doi:10.47102/annals-acadmedsg.2020459 · Cited by 21
Brothers, T. D., Lewer, D., Bonn, M., Webster, D., & Harris, M. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-049924 · Cited by 16
Coping in Post-Mastectomy Breast Cancer Survivors and Need for Intervention: Systematic Review.
Mishra, A., Nair, J., & Sharan, A. M. (2023) · Breast Cancer Basic and Clinical Research · doi:10.1177/11782234231209126 · Cited by 15
“11 The MMAT summarizing the various studies included is shown in Table 1.”
MethodBlamires, J., Annette, D., Napier, S., & Foster, M. (2023) · Children · doi:10.3390/children10061006 · Cited by 24
“Three authors independently assessed the quality of the included articles using the appropriate criteria from the MMAT [34].”
MethodInfluentialTalla, P. K., Inquimbert, C., Dawson, A., Zidarov, D., Bergeron, F., & Chandad, F. (2023) · JMIR Research Protocols · doi:10.2196/44218 · Cited by 24
“Two independent reviewers will assess the methodological quality of the included studies using the core quality criteria in the Mixed Methods Appraisal Tool (MMAT) [47].”
MethodInfluentialUrtubia-Herrera, V., Sánchez, M. V. N., Santos, A. M. P., Marcos, A. P., García-Gómez, A., Luís, E. O., & Bermejo‐Martins, E. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1304310 · Cited by 24
Gilbert, E., de Viggiani, N., Martins, J. D. S., Palit, T., Sears, J., Knights, D., Roulston, A., Turner, M., & Selman, L. E. (2024) · Palliative Medicine · doi:10.1177/02692163241242647 · Cited by 20
“28 The MMAT incorporates descriptive criteria for evaluating mixed-methods studies and has been widely tested for content validity.”
Vestal, E., Newman, S. D., & Phillips, S. M. (2024) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2024.05.001 · Cited by 21
Andrews, M., Cheema, B. S., & Siette, J. (2024) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2024.105535 · Cited by 20
Platelet-Rich Plasma in Aesthetic Dermatology: Current Evidence and Future Directions
Asubiaro, J., & Avajah, F. (2024) · Cureus · doi:10.7759/cureus.66734 · Cited by 15
Medication Errors in Saudi Arabian Hospital Settings: A Systematic Review
Tobaiqy, M. A., & MacLure, K. (2024) · Medicina · doi:10.3390/medicina60091411 · Cited by 33
“Each of the included studies was subject to critical appraisal using the MMAT tool (Hong et al., 2018) appropriate to the study design [22].”
MethodManaging organizational carbon neutrality: A systematic review
Boiral, O., Morin, G., Yuriev, A., & Talbot, D. (2024) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.3061 · Cited by 18
almoliky, M. A., Alkubati, S. A., Saleh, K., Alsaqri, S., Al-Ahdal, S. A., Albani, G. F., & Sultan, M. A. (2025) · BMC Nursing · doi:10.1186/s12912-025-02704-x · Cited by 21
Musie, M. R., Tagutanazvo, O. B., Sepeng, N. V., Mulaudzi, F. M., & Hlongwane, T. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06830-7 · Cited by 24
“Using MMAT, we appraised the methodological quality of five categories of research: qualitative research, randomised controlled trials, nonrandomised studies, quantitative descriptive studies and mixed methods studies [23].”
Petersen, J., Müller, H., & Melzer, M. (2025) · Health & Social Care in the Community · doi:10.1155/hsc/7605478 · Cited by 23
“Tus, the included studies were critically appraised by two authors using the Critical Appraisal Skills Program (CASP-Tools) [27], which specializes in diferent types of studies, and the Mixed Methods Appraisal Tool [28].”
Sharp, J., Kelson, J. N., South, D., Saliba, A. J., & Kabir, M. A. (2025) · Acta Astronautica · doi:10.1016/j.actaastro.2025.04.002 · Cited by 20
“To guide future research on this topic and to meet the aims of the review, the Mixed Methods Appraisal Tool (MMAT) [34] was used by two reviewers (JS, AK) to appraise the quality of the included studies.”
Draeger, L., Fleischmann-Struzek, C., Gehrke-Beck, S., Heintze, C., Thomas-Rüddel, D. O., & Schmidt, K. F. R. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12777-8 · Cited by 26
“As a structured critical appraisal of the retrieved articles, three checklists were used to assess study quality: the JBI Checklist for Analytical Cross-Sectional Studies [20] for quantitative studies, the CASP Checklist [21] for qualitative studies, and the MMAT [22] for mixed-methods studies.”
López, M. O., & Gómez‐Baya, D. (2025) · Children · doi:10.3390/children12060665 · Cited by 32
“For assessing the risk of bias of mixed methods studies included in the review, the Mixed Methods Appraisal Tool (MMAT) [27] was used.”
Syasyila, K., Kamaluddin, M. R., Nor, H. A. M., & Singh, P. S. J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327810 · Cited by 23
Pham, C. T., Nguyen, H. T., Le, H. H. T. C., Tran, N. Q. L., Quoc, D. K., Bui, V., Phung, H. N., Phung, D. T., & Chu, C. M.-Y. (2025) · Tropical Medicine & International Health · doi:10.1111/tmi.70045 · Cited by 27
“The quality of the included studies were assessed using the Mixed Methods Appraisal Tool (MMAT), version 2018 [22].”
Agyenim-Boateng, A., Cameron, H., & Mensah, A. B. B. (2021) · International Journal of Africa Nursing Sciences · doi:10.1016/j.ijans.2021.100326 · Cited by 18
Fu, X., Lovell, A. L., Braakhuis, A. J., Mithen, R. F., & Wall, C. R. (2021) · Nutrients · doi:10.3390/nu13114105 · Cited by 19
“2018) [28] was used by two reviewers (X.”
MethodRahman, S., Elliott, S., Scott, S., & Hartling, L. (2022) · PEC Innovation · doi:10.1016/j.pecinn.2022.100018 · Cited by 13
“We used the Mixed-method Appraisal Tool (MMAT) version 2018 to assess the methodological quality of the studies [31,32].”
MethodDonachie, K., Cornel, E., Pelgrim, T., Michielsen, L. J., Langenveld, B., Adriaansen, M., Bakker, E., & Lechner, L. (2022) · Supportive Care in Cancer · doi:10.1007/s00520-022-06830-z · Cited by 16
Xulu-Kasaba, Z. N., & Kalinda, C. (2022) · Tropical Medicine and Infectious Disease · doi:10.3390/tropicalmed7020034 · Cited by 18
Risk factors associated with 30‐day unplanned hospital readmission for patients with mental illness
Zhou, H., Ngune, I., Albrecht, M. A., & Della, P. R. (2022) · International Journal of Mental Health Nursing · doi:10.1111/inm.13042 · Cited by 15
Pharmacists’ naloxone services beyond community pharmacy settings: A systematic review
Rawal, S., Osae, S. P., Cobran, E. K., Albert, A., & Young, H. N. (2022) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2022.09.002 · Cited by 19
Orth, Z., & van Wyk, B. E. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.955869 · Cited by 16
Kristensen, J. Å., Skilbred, A., Abrahamsen, F. E., Ommundsen, Y., & Loland, S. (2022) · Performance Enhancement & Health · doi:10.1016/j.peh.2022.100237 · Cited by 28
Durojaiye, O. C., Jibril, I., & Kritsotakis, E. I. (2022) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x221131842 · Cited by 23
“We used the Mixed Methods Appraisal Tool (MMAT) version 2018 to access the methodological quality of the included studies, 16 due to differences in study design.”
Hossain, M. A., & Dawson, A. J. (2022) · WHO South-East Asia Journal of Public Health · doi:10.4103/who-seajph.who-seajph_144_21 · Cited by 22
Hughes, M. C., Afrin, S., & Hamlish, T. (2022) · Journal of Cancer Education · doi:10.1007/s13187-022-02236-x · Cited by 18
“We determined each study’s quality as high, medium, and low based on the questions and criteria in the Mixed Method Appraisal Tool for qualitative, quantitative, and mixed-method studies [17].”
MethodDelgado, J., Ramos‐García, V., Infante‐Ventura, D., Suárez-Herrera, J. C., Rueda‐Domínguez, A., Serrano‐Aguilar, P., & Trujillo‐Martín, M. M. (2022) · Health Expectations · doi:10.1111/hex.13679 · Cited by 14
“supported are the data from the primary studies and from the synthesized outcomes provided by reviews) and adequacy (the degree of richness and amount of data supporting a review conclusion).(23,26)”
MethodResultInfluentialCooper, K. B., Lapierre, S., Seoane, M. C., Lindstrom, K., Pritschmann, R. K., Donahue, M. L., Christou, D. D., McVay, M. A., & Jake‐Schoffman, D. E. (2023) · Translational Behavioral Medicine · doi:10.1093/tbm/ibac111 · Cited by 19
Dabi, G. K., & Golga, D. N. (2023) · British Journal of Visual Impairment · doi:10.1177/02646196231158922 · Cited by 15
“The mixed-methods appraisal tool was used to assess the quality of the selected (11) papers using (MMAT-version 2018).”
InfluentialWladkowski, S. P., Wallace, C. L., Coccia, K. W., et al. (2023) · American Journal of Hospice and Palliative Medicine® · doi:10.1177/10499091231168401 · Cited by 15
“In light of our decision to review quantitative, qualitative, and mixed methods studies, we chose a modi fi ed version of the Mixed Methods Appraisal Tool 14 (MMAT; Table 2) to guide our critical appraisal of the included studies ’ quality.”
MethodCoombs, C., Cohen, T., Duddy, C., Mahtani, K. R., Owen-Boukra, E., Roberts, N. W., Saini, A., Foster, A. S., & Park, S. E. (2023) · British Journal of General Practice · doi:10.3399/bjgp.2022.0545 · Cited by 21
Integrative oncology in cancer care – implementation factors: mixed-methods systematic review
Kwong, M. H., Ho, L. T. F., Li, A. S., Nilsén, P., Ho, F. F., Zhong, C. C., & Chung, V. C. (2023) · BMJ Supportive & Palliative Care · doi:10.1136/spcare-2022-004150 · Cited by 15
Brownie, S. M., Blanchard, D. L., Amankwaa, I., Broman, P., Haggie, M., Logan, C., Pearce, A., Sampath, K. K., Yan, A., & Andersen, P. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1124264 · Cited by 15
“Application of the the ‘MMAT version 2018 (36) provided the quality appraisal results shown in Figure 2.”
BackgroundMethodNewcomer, K. E., Hall, J. L., Pandey, S. K., Reginal, T., & White, B. P. (2023) · Public Administration Review · doi:10.1111/puar.13688 · Cited by 20
“…to appraise the rigor of mixed methods studies, we first drafted proposed standards based upon routinely cited criteria for both quantitative and qualitative research (Hendren et al., 2022; Hong et al., 2018; Newcomer et al., 2015; Newcomer & Hart, 2022; O ’ Cathain, 2010; Pluye et al., 2009).”
Klijn, N. F., ter Kuile, M. M., & Lashley, E. E. L. O. (2023) · Cancers · doi:10.3390/cancers15245828 · Cited by 16
“The quality of the selected studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [12].”
MethodGrauman, Å., Ancillotti, M., Veldwijk, J., & Mascalzoni, D. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02395-x · Cited by 20
“The Mixed Methods Appraisal Tool (MMAT) [23] was selected to evaluate the methodological quality of the included studies.”
MethodGordon, K., Gordon, L., & Basu, A. P. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-078097 · Cited by 16
Cooke, K., Ridgway, K., Pecora, L. A., Westrupp, E. M., Hedley, D., Hooley, M., & Stokes, M. A. (2023) · Research in Autism · doi:10.1016/j.rasd.2023.102313 · Cited by 30
Dutch, D. C., Bell, L., Zarnowiecki, D. M., Johnson, B. J., Denney‐Wilson, E., Byrne, R., Cheng, H., Rossiter, C., Manson, A., House, E. T., et al. (2024) · Obesity Reviews · doi:10.1111/obr.13694 · Cited by 16
Heathcote, C., Taylor, J., Hall, R., Jarvis, S. W., Langton, T., Hewitt, C. E., & Fraser, L. (2024) · Archives of Disease in Childhood · doi:10.1136/archdischild-2023-326347 · Cited by 24
Liu, S., & Noh, Y. (2024) · Australian Journal of Psychology · doi:10.1080/00049530.2024.2357627 · Cited by 15
“The methodological quality appraisal utilized the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MicroRNA Expression in Patients with Coronary Artery Disease and Hypertension—A Systematic Review
Kondracki, B., Kłoda, M., Jusiak-Kłoda, A., Kondracka, A., Wacinski, J., & Wacinski, P. J. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25126430 · Cited by 15
“The Mixed Methods Assessment Tool (MMAT) assessed paper quality for the systematic literature review, ensuring comprehensive evaluation of both qualitative and quantitative studies [8].”
MethodProviders’ experiences with abortion care: A scoping review
Dempsey, B., Callaghan, S. L., & Higgins, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303601 · Cited by 15
“We also conducted a quality appraisal on the included studies using the Mixed Methods Appraisal Tool [19].”
BackgroundMethodПраведников, А. В., Perkovic, S., & Lagerkvist, C. J. (2024) · Environmental Research · doi:10.1016/j.envres.2024.119836 · Cited by 20
Mental Health Stigma Reduction Interventions Among Men: A Systematic Review
Sweeney, J., O’Donnell, S., Roche, E., White, P. J., Carroll, P., & Richardson, N. (2024) · American Journal of Men s Health · doi:10.1177/15579883241299353 · Cited by 21
Vuoriainen, A., Rikala, J., Heilala, V., Lehesvuori, S., Öz, Ş., Kettunen, L., & Hämäläinen, R. (2024) · European Journal of Engineering Education · doi:10.1080/03043797.2024.2432440 · Cited by 42
“The methodological quality of the included studies was appraised by using the Mixed Methods Appraisal Tool (MMAT) by Hong et al. (2018).”
Stigma Toward Persons with Intellectual Disability in South America: A Narrative Review
Tenorio, M., Donoso, J., Ali, A., & Hassiotis, A. (2020) · Journal of Policy and Practice in Intellectual Disabilities · doi:10.1111/jppi.12357 · Cited by 19
Zivot, C., Dewey, C. E., Heasley, C., Srinivasan, S., & Little, M. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17207511 · Cited by 21
The antecedents of well‐being in first‐generation migrant children: A systematic review
Marcos, E. B., Serrano, I., & García, M. M. F. (2021) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12282 · Cited by 20
“To identify strengths and limitations of the selected primary studies, a quality assessment was made following the Mixed- Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018).”
MethodGarritsen, H. H., Senior, Y. D. C., Rozema, A. D., Kunst, A. E., & Kuipers, M. A. G. (2021) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntab129 · Cited by 15
What Is Intensity and How Can It Benefit Exercise Intervention in People With Stroke? A Rapid Review
Church, G., Smith, C., Ali, A., & Sage, K. (2021) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2021.722668 · Cited by 21
“The 17 articles were subjected to MMAT evaluation; eight were quantitative non-randomized controlled trials (RCTs), eight were quantitative RCTs, and two were qualitative reviews of the literature.”
MethodInfluentialAlbaum, C. S., Chan, V., Sellitto, T., Vashi, N., Hastings, R. P., & Weiss, J. A. (2021) · International review of research in developmental disabilities · doi:10.1016/bs.irrdd.2021.08.003 · Cited by 23
Domestic Homicide Review Committees’ Recommendations and Impacts: A Systematic Review
Jones, C. A., Bracewell, K., Clegg, A. J., Stanley, N., & Chantler, K. (2022) · Homicide Studies · doi:10.1177/10887679221081788 · Cited by 24
Electronic Discharge Communication Tools Used in Pediatric Emergency Departments: Systematic Review
Wozney, L., Curran, J., Archambault, P. M., Cassidy, C. E., Jabbour, M., Mackay, R., Newton, A. S., Plint, A. C., & Somerville, M. (2022) · JMIR Pediatrics and Parenting · doi:10.2196/36878 · Cited by 16
“As a broad range of study designs was anticipated, the Mixed Method Appraisal Tool (MMAT) version 2018 [25] was used for methodological quality appraisal.”
MethodSelf-Medication Among Pregnant Women in Ghana: A Systematic Review and Meta-Analysis
Opoku, R., Yar, D. D., & Botchwey, C. O. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e10777 · Cited by 23
Luijendijk, H. J., Page, M. J., Burger, H., & Koolman, X. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01115-7 · Cited by 20
Intergenerational interventions and their impact on active aging: A systematic review
Savino, M., De Luca, L., Nocentini, A., & Menesini, E. (2026) · International Psychogeriatrics · doi:10.1016/j.inpsyc.2025.100142 · Cited by 14
Impact of Patient Online Record Access on Documentation: Scoping Review
Meier-Diedrich, E., Lyckblad, C., Davidge, G., Hägglund, M., Kharko, A., McMillan, B., Blease, C., & Schwarz, J. (2025) · Journal of Medical Internet Research · doi:10.2196/64762 · Cited by 15
“The quality and methodological rigor of the studies were assessed using the Mixed Methods Appraisal Tool (MMAT) [34].”
Wijayaratnam, A., Kozłowska, O., Krayem, A., Kaur, S., Ayres, H., Smith, R., Paterson, J., Moghabghab, R., & Henshall, C. L. (2024) · International Journal of Mental Health Nursing · doi:10.1111/inm.13317 · Cited by 15
“Quality assessments of all included studies were completed independently by two researchers using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018), with the third researcher providing consensus.”
MethodImproving the quality of neonatal health care in Ethiopia: a systematic review
Belay, D. M., Erku, D., Bayih, W. A., Kassie, Y. T., Birhane, B. M., & Assefa, Y. A. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1293473 · Cited by 17
Medication administration in aged care facilities: A mixed‐methods systematic review
Garratt, S., Dowling, A., & Manias, E. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16318 · Cited by 15
“The Mixed-Methods Appraisal Tool facilitates the appraisal of studies in five overarching designs: randomized control trials, qualitative, mixed-methods, quantitative descriptive and non-randomized studies (Hong et al., 2018 ).”
Sadeghi, H., Shahboulaghi, F. M., Hosseini, M., Fallahi‐Khoshknab, M., & Harouni, G. G. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1412832 · Cited by 34
Asfani, K., & Chen, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13125-9 · Cited by 24
“MMATv2018 enables critical appraisal of studies applying qualitative, quantitative, and mixed-method approaches (Hong et al., 2018).”
Yi, T., Shreyans, P., & Vallabhajosyula, R. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06716-8 · Cited by 23
Interactive Conversational Agents for Perinatal Health: A Mixed Methods Systematic Review
Amil, S., Da, S. M. A. R., Plaisimond, J., Roch, G., Sasseville, M., Bergeron, F., & Gagnon, M. (2025) · Healthcare · doi:10.3390/healthcare13040363 · Cited by 40
“We used the Mixed Methods Appraisal Tool (MMAT) to assess the quality of all included studies [31].”
Self-citationShahsavar, Y., & Choudhury, A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319983 · Cited by 33
Ning, J., Ma, Z., Yao, J., Wang, Q., & Zhang, B. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2478437 · Cited by 27
“…have been assessed for quality using the Mixed Methods Appraisal Tool (MMAT), which is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, especially reviews that include qualitative, quantitative and mixed methods articles (Hong et al., 2018).”
Hu, A., Liu, Q., & Daniel, B. K. (2025) · SAGE Open · doi:10.1177/21582440251357198 · Cited by 59
“The Mixed Methods Appraisal Tool (MMAT) offers a systematic approach to evaluating studies across various research designs—quali-tative, quantitative, and mixed methods.”
Cangelosi, G., Conti, A., Caggianelli, G., Panella, M., Petrelli, F., Mancin, S., Ratti, M., & Masini, A. (2025) · Medicina · doi:10.3390/medicina61081403 · Cited by 31
“The risk of bias and methodological quality of the included studies were evaluated using JBI checklists for qualitative [45] and cross-sectional studies [46] and the MMAT tool for mixed-methods studies [47].”
Dey, A., Lewis, Z., Posel, J., Pan, R. Y., & Wang, K. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2025-301663 · Cited by 25
Machado, A., Ferreira, D. F., Ferreira, S., Almeida‐Antunes, N., Carvalho, P. J., Melo, P. I. D. A., Rocha, N., & Rodrigues, M. A. (2025) · Sensors · doi:10.3390/s25196049 · Cited by 22
“This review’s studies were evaluated using the Mixed Methods Appraisal Tool (MMAT) from 2018 [20].”
Making sense of conducting a critical interpretive synthesis: A scoping review
Perlman, S., Ben-Sheleg, E., & Ellen, M. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10041 · Cited by 29
Mosha, N. F., & Ngulube, P. (2025) · Discover Sustainability · doi:10.1007/s43621-025-01983-3 · Cited by 21
“As the review included only mixed methods studies, the Mixed Methods Appraisal Tool (MMAT), a version 2018 assessment tool [33], was used.”
Gandara, E., Harvey, I. S., Foster, M., Luo, W., McKyer, L. E., Burdine, J. N., & Martinez, D. (2022) · Journal of Religion and Health · doi:10.1007/s10943-022-01532-6 · Cited by 15
Self-care behaviors in patients with cancer treated with oral anticancer agents: a systematic review
Nitto, M. D., Sollazzo, F., Biagioli, V., Pucciarelli, G., Torino, F., Alvaro, R., & Vellone, E. (2022) · Supportive Care in Cancer · doi:10.1007/s00520-022-07166-4 · Cited by 17
“None of the 19 studies was excluded after quality appraisal, in line with the recommendations of the MMAT [37].”
MethodResultInfluentialAlrø, A. B., Nedergaard, H. K., Svenningsen, H., Jensen, H. I., & Dreyer, P. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16490 · Cited by 21
Lyu, R., Lu, S., & Ma, X. (2022) · Family Relations · doi:10.1111/fare.12768 · Cited by 22
Integrated nursing care for people with combined mental health and substance use disorders
Hove, E., Hazelton, M., Santangelo, P., & Wilson, R. L. (2022) · International Journal of Mental Health Nursing · doi:10.1111/inm.13094 · Cited by 26
Family-centered collaborative care for patients with chronic mental illness: A systematic review
Dehbozorgi, R., Shahriari, M., Moghadam, M. F., & Sarani, E. M. (2023) · Journal of Research in Medical Sciences · doi:10.4103/jrms.jrms_410_22 · Cited by 16
“We used the Verhagen quality appraisal tool[30] for randomized‑control trial (RCT) articles, Mixed Methods Appraisal Tool.”
Gelman, R., Whelan, J., Spiteri, S., Duric, D., Oakhill, W., Cassar, S., & Love, P. (2023) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-023-01433-1 · Cited by 21
Zavala, G. A., Jennings, H. M., Afaq, S., Alam, A., Ahmed, N., Aslam, F., Arsh, A., Coales, K., Ekers, D., Nabi, M., et al. (2023) · International Journal of Mental Health · doi:10.1080/00207411.2023.2202431 · Cited by 16
“Quality was scored as mostly poor on the Hawker et al. (2002) scale (Hawker et al., 2002), and mostly adequate on the MMAT (Hong et al., 2018).”
MethodLester, H., Ryakhovskaya, Y., & Olorunnisola, T. S. (2023) · International Journal of Migration Health and Social Care · doi:10.1108/ijmhsc-09-2022-0098 · Cited by 14
Lantta, T., Duxbury, J. A., Haines‐Delmont, A., Björkdahl, A., Husum, T. L., Lickiewicz, J., Douzenis, A., Craig, E., Goodall, K., Bora, C., et al. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1158145 · Cited by 19
“Quality appraisal of included studies was undertaken using MMAT—Mixed Methods Appraisal Tool (21).”
BackgroundMethodInfluentialPedagogical Models, Physical Activity and Intention to be Physically Active: A Systematic Review
Saiz-González, P., Gallego, D. I., & Fernández‐Río, J. (2023) · Quest · doi:10.1080/00336297.2023.2209734 · Cited by 24
“The Mixed Methods Assessment Tool (MMAT; Hong et al., 2018) was used.”
MethodDoes the feeding method affect the quality of infant and maternal sleep? A systematic review
Manková, D., Švancarová, S., & Štenclová, E. (2023) · Infant Behavior and Development · doi:10.1016/j.infbeh.2023.101868 · Cited by 16
Empathy training for service employees: A mixed-methods systematic review
Lajante, M., Prete, M. D., Sasseville, B., Rouleau, G., Gagnon, M., & Pelletier, N. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289793 · Cited by 48
Self-citationMercera, G., Kooijmans, R., Leijdesdorff, S. M. J., Heynen, E. J. E., & van Amelsvoort, T. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231201815 · Cited by 18
“The Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) was used to appraise the methodological quality of the included studies.”
MethodMuluh, E. A. E., McCormack, J. C., Mo, Y., Garratt, M., & Peng, M. (2023) · Physiology & Behavior · doi:10.1016/j.physbeh.2023.114388 · Cited by 17
“We rated each criterion as "yes, " "no," or "cannot tell" based on the information provided in the study [43].”
MethodImplementation and Impact of Health Care Gardens: A Systematic Scoping Review
Carroll, C., Higgs, J., McCray, S. J., & Utter, J. (2023) · Journal of Integrative and Complementary Medicine · doi:10.1089/jicm.2023.0212 · Cited by 19
“26 The MMAT is a validated critical appraisal tool designed to assess multiple study designs, including qualitative research, randomized control trials, nonrandomized control trials, quantitative descriptive studies, and mixed method studies.”
InfluentialYoga nidra for mental health: A systematic review of current evidence
Malviya, S., Saoji, A. A., & Pravalika, B. (2023) · Journal of Spirituality in Mental Health · doi:10.1080/19349637.2023.2290249 · Cited by 14
“Of 16 included studies, six met all the criteria of the quality assessment scale (D’cunha et al., 2021; D’souza, Jose, Suresh, & Baliga, 2021; Dol, 2019; Studies that scored less than three on the MMAT quality assessment Moszeik, von Oertzen, & Renner, 2022; Stankovic, 2011; Vaishnav, Vaishnav, Vaishnav, & Varma, 2018…”
InfluentialClark, N., Kainth, G. S., Johnson, M., Rangan, A., Kottam, L., & Swainston, K. (2023) · Rheumatology International · doi:10.1007/s00296-023-05503-2 · Cited by 19
“The risk of bias was assessed using the Mixed Methods Assessment Tool [26] as it allows for the assessment of varied study designs, as included in this review.”
MethodFoo, B. M. Y., Sharpe, L., Clayton, J. M., Wiese, M., & Menzies, R. E. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102393 · Cited by 15
Murden, R., Allan, S. M., Hodgekins, J., & Oduola, S. (2024) · Schizophrenia Research · doi:10.1016/j.schres.2024.02.032 · Cited by 14
“MMAT contains two generic questions measuring quality, followed by five further questions depending on the study method.”
InfluentialGray, S. M., McMorris, C. A., Mudry, T., & McCrimmon, A. W. (2024) · Research in Autism · doi:10.1016/j.rasd.2024.102394 · Cited by 20
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.