The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,201–2,300 of 4,578
Peters-Mosquera, A., Bingham-Hendricks, C., Woods, C., & Aronowitz, T. B. (2023) · Journal of Transcultural Nursing · doi:10.1177/10436596231198274 · Cited by 4
“We used the MMAT to evaluate the studies (Hong et al., 2018; Pluye et al., 2011), which encourages a descriptive quality appraisal rather than a numerical score.”
MethodClark, A. M., Sousa, B. J., Ski, C. F., Redfern, J., Neubeck, L., Allana, S., Peart, A., MacDougall, D., & Thompson, D. R. (2023) · Journal of Cardiopulmonary Rehabilitation and Prevention · doi:10.1097/hcr.0000000000000802 · Cited by 6
“22,23 These assess elements of study design, recruitment, and methods.”
BackgroundAlarifi, S., Denne, L., Alatifi, N. M., & Hastings, R. P. (2024) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.13205 · Cited by 8
“Another reason for selecting the MMAT is that its validity and reliability have been established (Pace et al., 2012), and it has already been used in several reviews (e.g., Steel et al., 2020; Xu et al., 2021).”
InfluentialWhat works to promote staff health in prison settings: a systematic review
Woodall, J. (2024) · International Journal of Prison Health · doi:10.1108/ijoph-11-2023-0082 · Cited by 7
A systematic review of the factors associated with malaria infection among forest rangers
Dapari, R., Yusop, M. Z. F. M., Chinnasamy, D., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303406 · Cited by 4
“During the assessment, the quality of each article was evaluated based on methodological criteria, including five core quality parameters [13].”
Nxele, S. R., Moetlhoa, B., Dlangalala, T., Maluleke, K., Kgarosi, K., Theberge, A. B., & Mashamba-Thompson, T. P. (2024) · Archives of Public Health · doi:10.1186/s13690-024-01376-4 · Cited by 4
“We evaluated the quality of the included articles using the mixed method appraisal tool [24] version 2018 [25].”
MethodMunhoz, O. L., Morais, B. X., da Luz, E. M. F., Greco, P. B. T., Ilha, S., & Magnago, T. S. B. D. S. (2024) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-0347pt · Cited by 6
Humayra, S., Yahya, N. A., Ning, C. J., Mir, I. A., Mohamed, A. L., & Manan, H. A. (2024) · Frontiers in Neurology · doi:10.3389/fneur.2024.1425399 · Cited by 7
Professional Bereavement in Nursing: An Evolutionary Concept Analysis
He, X., Wang, Y., Yang, C., Cao, H., Wang, Q., & Hui, Y. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16545 · Cited by 4
Armitage, L., Atchan, M., Davis, D. L., Turner, M. R., & Paterson, C. C. (2024) · Journal of Cancer Survivorship · doi:10.1007/s11764-024-01695-z · Cited by 6
“The methodological quality and evaluation of the studies were assessed using the Mixed Methods Appraisal Tool (MMAT) [14] as it enables qualitative, quantitative, and mixed method research designs to be evaluated using the same form which increases consistency of findings.”
E‐Professionalism's Impact on Dental Professionalism: A Rapid Scoping Review
Nasseripour, M., Harkins, A., Neville, P., & Ranauta, A. (2024) · European Journal Of Dental Education · doi:10.1111/eje.13051 · Cited by 6
van Alem, J. L. L., Frielink, N., & Embregts, P. J. C. M. (2025) · Journal of Intellectual Disability Research · doi:10.1111/jir.13211 · Cited by 12
“The risk of bias was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al. 2018).”
Effects of School Nurse-Led Asthma Interventions for Students: A Systematic Review
Yoon, J., Seo, H., Cho, J., Lee, S. J., Lee, J. S., Choi, Y., & Noh, S. (2025) · The Journal of School Nursing · doi:10.1177/10598405241311464 · Cited by 5
“To assess the risk of bias, we used the Mixed Methods Appraisal Tool (MMAT) 2018 version, a widely accepted tool for appraising the methodological quality of literature review involving diverse study designs (Hong et al., 2018).”
Cai, Y., Delaney, L., Lynch, D., Cunninghame, J., Takashima, M., Ullman, A. J., Toombs, M. R., Martin, A., Thompson, K., Maxwell, K., et al. (2025) · First Nations Health and Wellbeing - The Lowitja Journal · doi:10.1016/j.fnhli.2024.100040 · Cited by 4
Aldakhil, R., Greenfield, G., Lammila-Escalera, E., Laranjo, L., Hayhoe, B. W. J., Majeed, A., & Neves, A. L. (2025) · Journal of Diabetes Science and Technology · doi:10.1177/19322968251316585 · Cited by 8
A systematic review of the relationships amongst older adults’ cognitive and motor speech abilities
Manderson, L., Krzeczkowska, A., Kuschmann, A., Lowit, A., & Brown, L. A. (2025) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2025.106510 · Cited by 6
Transição de cuidados hospital-domicílio: estudo de métodos mistos à luz da Teoria de Meleis
Gama, L. M. D. P., Ribeiro, K. R. C., de Oliveira, J. L. C., et al. (2025) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2023-0357pt · Cited by 4
Indoor Environmental Monitoring and Chronic Respiratory Diseases: A Systematic Review
Camacho-Magriñán, P., Sales-Lérida, D., León‐Jiménez, A., & Morillo, D. (2025) · Technologies · doi:10.3390/technologies13030122 · Cited by 6
Gharagozloo, M., Moridi, M., Alimardi, M., & Moghadam, Z. B. (2025) · European journal of medical research · doi:10.1186/s40001-025-02458-x · Cited by 5
“Appraisal Tool (MMAT), a valid and reliable tool designed for use in systematic mixed studies reviews [19].”
Factors influencing health-promoting lifestyle among medical personnel: a systematic review protocol
Huang, P., Abai, D. S. A., Xiao, H., Zhang, Q., Xian, Z., & Abdullah, K. B. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-097470 · Cited by 5
Iqbal, F. M., Kayikci, S., Lowther-Payne, H. J., et al. (2025) · BMC Public Health · doi:10.1186/s12889-025-22582-x · Cited by 4
“MMAT has been used in a number of systematic reviews, and can be utilised in assessing the quality of studies with various designs applicable in this study [16].”
Kennedy, S., Fuller, P., Cha, J. S., Carbonell, A. M., Luo, Q., & Joseph, A. (2025) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208251333907 · Cited by 6
“To address this, any articles in question were brought to the larger team for review, and a second independent reviewer was used for the MMAT quality review of the articles to ensure all articles included were relevant and evaluated twice for quality.”
InfluentialDigital technology for supporting Aged Care Services: A Scoping Review
Theopilus, Y., Davis, H., & Pedell, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713498 · Cited by 14
“Some appraisal tools have been developed to assess the risk of bias in research with multiple types of studies, such as the Mixed Methods Appraisal Tool (MMAT) [40], Crowe Critical Appraisal Tool (CCAT) [41], and Quality Assessment with Diverse Studies (QuADS) [42].”
Zhou, W., Liang, Q., Bian, Y., Liu, Q., Li, Y., Yu, J., Chen, H., & Wang, C. (2025) · Journal of Clinical Nursing · doi:10.1111/jocn.17799 · Cited by 8
Sundus, A., Younas, A., & Gıwa, S. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.17015 · Cited by 4
Choi, H. R., Ho, M., & Lin, C. (2025) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2025.104037 · Cited by 7
A methodological review of patient healthcare-seeking journeys from symptom onset to receipt of care
Omenka, C. O., Sassi, A., Vasquez, N. A., Rana, N., Essar, M. Y., Ku, D., Diploma, H., Huria, L., Saqib, K., Das, R., et al. (2025) · BMJ Global Health · doi:10.1136/bmjgh-2024-016978 · Cited by 8
“The data extraction form aimed to collect general”
Petinger, C., Crowley, T., & van Wyk, B. E. (2025) · Adolescents · doi:10.3390/adolescents5020021 · Cited by 4
Kharko, A., Hägglund, M., Angelova, D. M., Duncan, T. S., Hagström, J., Hansford, K. J., Hunt, J., Lagerros, Y. T., Locher, C., McMillan, B., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-098241 · Cited by 6
“28 The MMAT was chosen as it will enable consistent appraisal across a variety of study designs.”
Kassab, S. E., Rathan, R., & Hamdy, H. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02442-4 · Cited by 5
“For mixed-methods studies, it assesses integration and the justification for using a mixed design [34].”
InfluentialO’Reilly, P., Savarimalai, R., Walsh, S., et al. (2025) · British Journal of Dermatology · doi:10.1093/bjd/ljaf229 · Cited by 4
Lee, H., Shin, Y., Moon, H., Choi, Y., & Lee, A. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100850 · Cited by 8
“The MMAT-based quality assessment revealed that the majority of included studies demonstrated high methodological quality, with 16 studies achieving a perfect score and 33 scoring above 60 %.”
InfluentialGelner, H., Krężołek, M., Aleksandrowicz, A., Piwińska, J., Lamarca, M., Berna, F. S., Vanessa Acuña, P., König, C., Moritz, S., Ochoa, S., et al. (2025) · Schizophrenia Research · doi:10.1016/j.schres.2025.06.017 · Cited by 6
Kobson, B., Hanley, J., Mair, A. R., Dima, A. L., Rea, N., & Paterson, R. (2025) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-025-01958-4 · Cited by 8
“Quality appraisal was undertaken using the Mixed Methods Appraisal Tool (MMAT) [33], which was selected due to its suitability for assessing the methodological quality of studies with diverse designs, including qualitative, quantitative, and mixed-methods approaches.”
Creighton, L., Caughers, G., Mitchell, G., McMahon, J., Neubeck, L., & Fitzsimons, D. (2025) · BMC Nursing · doi:10.1186/s12912-025-03611-x · Cited by 10
“In this review, methodological quality and risk of bias was done on the included empirical research articles using the Mixed Methods Appraisal Tool (MMAT) to identify high, medium and low quality of evidence [40].”
Sutton, K. N., Lydon, E., Rader, A., Barton, K. D., Skalla, L. A., Wang, T.-W., Ruffin, F., Oakes, M. M., Miller, J., Collyar, D. E., et al. (2025) · Clinical Infectious Diseases · doi:10.1093/cid/ciaf294 · Cited by 4
Leese, C. J., Abraham, K., van de Konijnenburg, C., & Al-Zubaidi, H. (2025) · Primary Health Care Research & Development · doi:10.1017/s1463423625100339 · Cited by 5
“Study quality was assessed using the Mixed Methods Appraisal Tool (MMAT) Version 2018, accounting for the wide range of study types included (Hong et al ., 2018).”
Public health risks of traditional zootherapeutic practices in Africa
Fourchault, L., Lamane, A., Romaric, N. M. D., Saliu, G. T., Gryseels, S., Verheyen, E., Dahdouh‐Guebas, F., & Kreppel, K. S. (2025) · One Health · doi:10.1016/j.onehlt.2025.101178 · Cited by 7
“Following quality analysis using the MMAT framework [24] (Additional file 1), studies were included if they were published in peer-reviewed journals or as scientific theses in French or English after 1990, and if they explicitly stated the name of the animal and the ailment treated, the animal…”
Lörzing, R., & Bartelmeß, T. (2025) · Nutrition & Dietetics · doi:10.1111/1747-0080.70046 · Cited by 8
“26 The first author conducted the quality assessment.”
Mapping the ecological landscape of early childhood creativity: A decade of empirical research
Xu, Y., Wang, Q., & Li, C. (2025) · Cognitive Development · doi:10.1016/j.cogdev.2025.101650 · Cited by 5
Groen, A., Lucas, C., Benson, H., Alsubaie, M. G., & Boyd, M. (2020) · Pharmacy Education · doi:10.46542/pe.2020.201.313323 · Cited by 5
“Figure A: PRISMA flow diagram of search strategy and study selection 21 Figures The Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) is the only tool that has established validity and reliability, enabling critical appraisal of all research designs to facilitate the synthesis of qualitative…”
MethodInfluentialMousavi, M. S. S., Khodayari‐Zarnaq, R., & Hajizadeh, A. (2021) · Clinical Ethics · doi:10.1177/1477750921994278 · Cited by 4
“The MMAT is intended to be used as a checklist for concomitantly appraising and/or describing studies included in systematic mixed studies’ reviews (reviews including original qualitative, quantitative, and mixed-method studies).(25) Based on the scores achieved from the assessment, the quality of articles was classif…”
MethodRumping, S., Boendermaker, L., & Fukkink, R. G. (2022) · Children and Youth Services Review · doi:10.1016/j.childyouth.2022.106365 · Cited by 4
Hammink, J. H. W., Moor, J. A., & Mohammadi, M. (. (2021) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2021.1994032 · Cited by 5
Clarke, D., Williamson, G. R., & Stebbings, A. (2021) · The Open Nursing Journal · doi:10.2174/1874434602115010273 · Cited by 6
“Therefore the Mixed Methods Appraisal Tool (MMAT) [26] and the Centre for Evidence Based Medicine (CEBM) [27] Critical Appraisal of a Survey tool were for those study designs.”
MethodOnline teaching effectiveness in higher education: Faculty perspectives in India
Mishra, L., & Raina, R. L. (2021) · Journal of University Teaching and Learning Practice · doi:10.53761/1.18.8.5 · Cited by 4
Risk factors for the recurrence of venous leg ulcers in adults: A systematic review protocol
Johnston, S., Finlayson, K. J., Bui, U. T., O'Donoghue, E., Fletcher, B., & Parker, C. N. (2022) · Journal of Tissue Viability · doi:10.1016/j.jtv.2022.06.006 · Cited by 6
Konstantinidis, K., & Apostolakis, I. (2022) · European Journal of Engineering and Technology Research · doi:10.24018/ejeng.2022.1.cie.2959 · Cited by 4
“MMAT permits assessment of the methodological quality in qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies [7].”
BackgroundNxele, S. R., Moetlhoa, B., Kgarosi, K., & Mashamba-Thompson, T. P. (2023) · PLoS ONE · doi:10.1371/journal.pone.0276827 · Cited by 4
“The quality of the included articles will be evaluated using the mixed method appraisal tool [23] version 2018 [24].”
MethodBrodeur, M., Roberge, P., Cotton, J.-C., Monson, E., Morvannou, A., Poitras, M.-È., Lacasse, A., Jutras‐Aswad, D., Couturier, Y., Loignon, C., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-066231 · Cited by 4
Job stress among Iranian firefighters: a systematic review
Mousavipour, S. S., Sheikhbardsiri, H., Golitaleb, M., Farahi-Ashtiani, I., Yousefi, K., & Sahebi, A. (2022) · Disaster and Emergency Medicine Journal · doi:10.5603/demj.a2022.0027 · Cited by 4
School Health Policies in the World: A Scoping Review
Asadi, Z., Teymourlouy, A. A., & Hashjin, A. A. (2023) · Health Scope · doi:10.5812/jhealthscope-133708 · Cited by 7
Sleep Health Among Adults in Outpatient Opioid Use Disorder Treatment: A Systematic Review
Hernández, E., & Griggs, S. A. (2023) · Journal of Psychosocial Nursing and Mental Health Services · doi:10.3928/02793695-20230622-04 · Cited by 5
Durojaiye, O. C., Cole, J., & Kritsotakis, E. I. (2023) · International Journal of Antimicrobial Agents · doi:10.1016/j.ijantimicag.2023.106911 · Cited by 6
Centenaro, A. P. F. C., da Silva, R. M., Franco, G. P., et al. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6669.4002 · Cited by 4
Biomarkers of Waterpipe Tobacco Smoke Exposure: A Systematic Review and Meta-Analysis
Yan, Y., Ye, F., Ho, M., Yeung, K. C. Y., & Lee, J. J. (2023) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntad262 · Cited by 4
Mediators of Outcome in Trauma-Focused Psychotherapy with Youth: A Systematic Review
Protić, S., Wittmann, L., Taubner, S., Cerón, S. C., Ioannou, Y., Heinonen, E. O., Saliba, A., Moreno-Peral, P., Volkert, J., Barkauskienė, R., et al. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380231223264 · Cited by 5
“This sums up to a maximum of six points for mediator study quality and also a maximum of six points for the overall study quality with the Mixed Methods Appraisal Tool.”
Physical activity interventions among youth living in rural and remote areas: A systematic review
Dennehy, J., Cameron, M., Phillips, T. M., & Kolbe-Alexander, T. L. (2024) · Australian and New Zealand Journal of Public Health · doi:10.1016/j.anzjph.2024.100137 · Cited by 9
Harris, J., Nowland, R. A., & Todd, M. (2024) · Journal of Educational Research and Practice · doi:10.5590/jerap.2024.14.1.05 · Cited by 10
“All eight studies were quality assessed using the Mixed Method Assessment Tool (Hong et al., 2018).”
MethodConceptualisation and implementation of integrated disease surveillance globally: a scoping review
Flodgren, G., Bezuidenhoudt, J. E., Alkanhal, N., Brinkwirth, S., & Lee, A. C. (2024) · Public Health · doi:10.1016/j.puhe.2024.02.018 · Cited by 6
Goulding, R., Goodwin, J., Saab, M. M., McCarthy, M. J., Happell, B., & O’Donovan, Á. (2024) · International Journal of Transgender Health · doi:10.1080/26895269.2024.2333519 · Cited by 6
“The Mixed-Methods Appraisal Tool (MMAT) was used to assess the methodological quality of the included studies (Hong et al., 2018).”
Exploring self-experience practices in dementia care: A scoping review
Wittmann, J., Bieber, A., Carroll, J., Forristal, K., Hopper, L., Janssen, N., Meyer, G., Riello, M., de Vugt, M. E., & Bauernschmidt, D. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302929 · Cited by 5
Mamolis, G., Triantafyllopoulou, P., & Jones, K. (2024) · Journal of Long-Term Care · doi:10.31389/jltc.189 · Cited by 6
Elliott, S. A., Rahman, S., Scott, S. D., Craig, W., Knisley, L., Shearer, K., & Hartling, L. (2024) · Open Access Emergency Medicine · doi:10.2147/oaem.s450191 · Cited by 7
Hamed, A. M., Moralejo, D. G., Pike, A. D., & Curran, V. R. (2024) · Journal of Professional Nursing · doi:10.1016/j.profnurs.2024.06.019 · Cited by 3
Spijker, J., Barlın, H., Grad, D. A., Gu, Y., Kļaviņa, A., Yaylagül, N. K., Kulla, G., Orhun, E., Ševčíková, A., Unim, B., et al. (2024) · JMIR Research Protocols · doi:10.2196/59900 · Cited by 5
“In addition, we will assess the quality of studies using the Mixed Methods Appraisal Tool [31], whereby any discrepancies will be resolved through consensus between the reviewers.”
MethodMidwifery education in Bangladesh: An in-depth analysis through a systematic review
Fraser, R., Downer, T., & Oprescu, F. (2024) · Women and Birth · doi:10.1016/j.wombi.2024.101661 · Cited by 6
Crossfield, C., Ozavci, G., Digby, R., et al. (2024) · Australian Critical Care · doi:10.1016/j.aucc.2024.06.013 · Cited by 4
Flynn, R. J., Pringle, A., & Roscoe, C. M. P. (2024) · Children · doi:10.3390/children11091066 · Cited by 4
“Study quality was assessed using the mixed-methods assessment tool (MMAT) and is reported using asterisks (*) to describe the quality rating of the article, ranging from 1*, where 20% of the quality criteria have been met, to 5*****, where 100% of the quality criteria have been met [79].”
MethodRibeiro, A. S., Husson, O., Reuvers, M. J. P., et al. (2024) · Health Policy and Technology · doi:10.1016/j.hlpt.2024.100915 · Cited by 4
“The review was conducted in adherence to the registered protocol and selected articles were critically appraised by reviewers AR and MR independently using the Mixed Methods Appraisal Tool (MMAT) [15].”
What Evidence Exists to Support Palliative Allied Health Practice in Aged Care: A Scoping Review
Farrer, O. D., & Tieman, J. (2024) · Healthcare · doi:10.3390/healthcare12191973 · Cited by 4
“Data extraction was completed by one reviewer (OF) and appraised using the Mixed Methods Appraisal Tool [11] (MMAT), selected to suit the varied study designs of the included papers.”
MethodHuddlestone, L., Shoesmith, E., Pervin, J., Stevens, R., Gilbody, S. M., & Ratschen, E. (2024) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntae237 · Cited by 6
“In the revised MMAT, 30 using an overall numerical score is discouraged, with Hong and colleagues favoring a detailed presentation of the ratings against the MMAT criteria to reflect the quality of the included studies.”
Ritual Child Homicide in Contemporary Africa: A Systematic Review of the Empirical Literature
Owusu, E. S. (2024) · International Criminal Justice Review · doi:10.1177/10575677241282014 · Cited by 5
“Hong et al. (2018b) observe that systematic reviews utilizing empirical research are exceptionally useful for examining complex phenomena and interventions.”
InfluentialGamlath, I., Buchanan‐Hagen, S., Alexander, L., & Searby, A. (2024) · International Journal of Mental Health Nursing · doi:10.1111/inm.13454 · Cited by 5
Humans: An integrative review exploring dehumanisation in advanced dementia
Bøgmose, J., Bøgeskov, B. M. O., Dening, T., & Martinsen, B. (2024) · Nursing Ethics · doi:10.1177/09697330241277989 · Cited by 5
“40 None of the texts were excluded by this quality appraisal, and 26 texts were eligible for analysis consisting of eleven qualitative and four quantitative research articles, two dissertations, one ethnography, and eight discussion papers.”
Scoping review about well-being in the ‘brain migration’ studies
Vega-Muñoz, A., González-Gómez-del-Miño, P., & Salazar-Sepúlveda, G. (2024) · MethodsX · doi:10.1016/j.mex.2024.103068 · Cited by 6
“The risk of bias of the included studies will be assessed according to Campbell et al. [ 37 ] in the case of theoretical articles and the Mixed Methods Appraisal Tool (MMAT) scale will be used for quantitative, qualitative, and mixed studies [ 38 ].”
Cardiovascular Health Disparities in Adults of Color Aged 18 to 50: A Systematic Review
Howard, Q., Adeyinka, S., Martin, S., & Griggs, S. A. (2024) · The Journal of Cardiovascular Nursing · doi:10.1097/jcn.0000000000001165 · Cited by 5
Academic success in undergraduate nursing education: An integrative review
Sharda, H. K., & Nowell, L. S. (2024) · Nurse Education Today · doi:10.1016/j.nedt.2024.106540 · Cited by 10
Schwerdtle, P. N., Moo, E., Pae, C., Shinee, E., Wegerdt, J., Ali, A., Thondoo, M., & Bowen, K. (2024) · The Journal of Climate Change and Health · doi:10.1016/j.joclim.2024.100376 · Cited by 8
Extended reality as a modality to train non-technical skills in healthcare: A scoping review
Gonzales, A., Jackson, C. D., & Cha, J. S. (2025) · Applied Ergonomics · doi:10.1016/j.apergo.2024.104463 · Cited by 5
Jang, M., Park, J., & Han, J. (2025) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2025.102787 · Cited by 4
Video Head Impulse Test in Children—A Systematic Review of Literature
Dasgupta, S., Mukherjee, A. L., Crunkhorn, R., Dawabah, S., Aslıer, N. G. Y., Ratnayake, S., & Manzari, L. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14020369 · Cited by 4
Morimoto, H., Fukudome, T., Tajima, A., & Noguchi, D. (2025) · Psychogeriatrics · doi:10.1111/psyg.13243 · Cited by 5
“The Mixed Methods Appraisal Tool (MMAT) 17 was used to assess the quality of the eligible studies.”
Review article: Coaching in emergency medicine: A systematic review and future research agenda
Rixon, A., Wilson, S. G., Wong, L. Y., & Elder, E. G. (2025) · Emergency Medicine Australasia · doi:10.1111/1742-6723.70001 · Cited by 5
Zeffiro, V., Tempesta, S., Morandini, R., Alvaro, R., Vellone, E., Pucciarelli, G., & D’Agostino, F. (2025) · International Journal of Nursing Knowledge · doi:10.1111/2047-3095.70001 · Cited by 8
Philip, V. S., Geetha, S., & Abraham, A. N. (2025) · Augmentative and Alternative Communication · doi:10.1080/07434618.2025.2458861 · Cited by 4
“the studies included in the review were evaluated for their quality using the Mixed Methods appraisal tool (MMAt), version 2018 (Hong et al., 2018).”
Alkhurayji, K. S., Alsuhaimi, A., Alangari, H., Al-Rayes, S., & Alumran, A. K. (2025) · Healthcare · doi:10.3390/healthcare13060580 · Cited by 10
Petersson‐Bloom, L., Ewe, L. P., Klefbeck, K., & Leifler, E. (2025) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-025-00507-z · Cited by 9
Pre-service teachers’ attitudes toward gameful practices – a systematic review
Bacsa-Károlyi, B., & Fehérvári, A. (2025) · European Journal of Teacher Education · doi:10.1080/02619768.2025.2505020 · Cited by 4
Harkema, L., Vriens, D. J., Vermeerbergen, L. J. G., Gerritsen, D. L., & Vermeulen, P. A. M. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12904-5 · Cited by 11
“Three quality appraisal tools were selected to assess the various study methodologies: the Mixed Meth-ods Appraisal Tool (MMAT) for empirical studies [37], Joanna Briggs Institute (JBI) checklist for systematic reviews [38], and the JBI checklist for text and opinion [39].”
Factors influencing nurses’ decision to blow the whistle: a systematic review
Elsehrawy, M. G., Ramadan, O. M. E., Eweida, R. S., et al. (2025) · BMC Nursing · doi:10.1186/s12912-025-03400-6 · Cited by 4
“MMAT helps to introduce a standardized method for the evaluation of research quality, such as qualitative, quantitative, and mixed studies [26].”
Muñoz-Urtubia, N., Vega-Muñoz, A., Salazar-Sepúlveda, G., García-Gordillo, M. Á., & Carmelo-Adsuar, J. (2025) · Frontiers in Pediatrics · doi:10.3389/fped.2025.1449436 · Cited by 9
A scoping review of transformational leadership development in health-related PhD programs
Löfflad-Bürkin, B. M., Matthiä, A., Krasna, H., Künzli, N., Bohlius, J., & Czabanowska, K. (2025) · Health Policy · doi:10.1016/j.healthpol.2025.105411 · Cited by 8
Delaney, A. L., Flatt, A., Koepp, H., et al. (2025) · American Journal of Speech-Language Pathology · doi:10.1044/2025_ajslp-25-00067 · Cited by 4
“The Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) was used to analyze the quality of the included studies.”
Quist-Therson, R., Kuupiel, D., & Hlongwana, K. W. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033341 · Cited by 4
Maphosa, T. P., Mulqueeny, D. M., Osei, E., et al. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01309-6 · Cited by 4
Fänge, A. M., Thordardottir, B., Ya-Nyonge, M. A., & Lethin, C. (2020) · Healthcare · doi:10.3390/healthcare8030240 · Cited by 5
GANBAT, D., Jugder, B., Ganbat, L., Tomoeda, M., Dungubat, E., Takahashi, Y., Mori, I., Shiomi, T., & Tomita, Y. (2021) · Current Cancer Drug Targets · doi:10.2174/1568009621999210120182834 · Cited by 4
Takhdat, K., Lamtali, S., & Adib, A. R. E. (2021) · Nurse Education Today · doi:10.1016/j.nedt.2021.105082 · Cited by 6
Survivorship Care Plan Preferences and Utilization Among Asian American Breast Cancer Survivors
Nolan, T. S., Arthur, E. K., Nwodim, O., et al. (2021) · Oncology nursing forum · doi:10.1188/21.onf.507-521 · Cited by 4
“The Mixed Methods Appraisal Tool was used to evaluate mixed-methods studies (Hong et al., 2018).”
Method
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.