The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 501–600 of 4,578
Tohănean, D. I., Vulpe, A.-M., Mijaică, R., & Alexe, D. I. (2025) · Applied Sciences · doi:10.3390/app15179826 · Cited by 77
Midwifery students’ experiences of bullying and workplace violence: A systematic review
Capper, T. S., Muurlink, O. T., & Williamson, M. J. (2020) · Midwifery · doi:10.1016/j.midw.2020.102819 · Cited by 38
Ali, H. M., Ranse, J., Roiko, A. H., & Desha, C. (2022) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x2200108x · Cited by 52
Raine, G., Evans, C., Uphoff, E., Brown, J. V. E., Crampton, P. E. S., Kehoe, A., Stewart, L. A., Finn, G. M., & Morgan, J. E. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060281 · Cited by 40
Parikh, P., Aparo, N. O., Nordhagen, S., & De Steur, H. (2022) · Food Research International · doi:10.1016/j.foodres.2022.111376 · Cited by 32
Legge, H., Toohey, K., Kavanagh, P. S., & Paterson, C. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-022-01275-z · Cited by 31
“The quality appraisal of all included studies was conducted by utilising the Mixed Methods Appraisal Tool (MMAT) [18].”
MethodInfluentialFu, R., Leff, S. S., Carroll, I. C., Brizzolara-Dove, S., & Campbell, K. (2022) · School Psychology Review · doi:10.1080/2372966x.2022.2128601 · Cited by 42
“In total, 54 publications met the inclusion criteria and were then reviewed following Joanna Briggs Institute (JBI) Critical Appraisal guidelines (The Joanna Briggs Institute, 2022) and Mixed Methods Appraisal Tool (Hong et al., 2018).”
MethodKonlan, K. D., & Shin, J. (2022) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001085 · Cited by 46
“Two researchers independently assessed the quality of each study using MMAT version 2018 [15].”
MethodResilience-enhancing interventions for family caregivers: A systematic review
Chi, N., Han, S., Lin, S., Fu, Y., Zhu, Z., Nakad, L., & Demiris, G. (2023) · Chronic Illness · doi:10.1177/17423953231174928 · Cited by 50
Chan, D. N. S., Li, C., Law, B. M. H., Choi, K. C., Lee, P. P. K., & So, W. K. W. (2023) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2023.100279 · Cited by 35
Exploring facilitation in virtual simulation in nursing education: A scoping review
Heyn, L. G., Brembo, E. A., Byermoen, K. R., Cruaud, C., Eide, H., Flo, J., Nordsteien, A., Overgaard, G., & Egilsdottir, H. Ö. (2023) · PEC Innovation · doi:10.1016/j.pecinn.2023.100233 · Cited by 43
“The quality of the included studies was assessed independently using the Mixed methods appraisal tool (MMAT) [22].”
MethodAtnafu, D. D., Khatri, R. B., & Assefa, Y. A. (2024) · Health Research Policy and Systems · doi:10.1186/s12961-023-01094-3 · Cited by 54
“An MMAT tool was used to assess the methodological quality of mixed-methods studies [26].”
BackgroundMethodExploring Lived Experiences of Receiving a Diagnosis of Autism in Adulthood: A Systematic Review
Nayyar, J. M., Stapleton, A., Guérin, S., & O’Connor, C. (2025) · Autism in Adulthood · doi:10.1089/aut.2023.0152 · Cited by 47
Racz, J. I., Bialocerkowski, A., Calteaux, I. G., & Farrell, L. J. (2024) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-024-00478-3 · Cited by 45
“Given the wide array of designs deemed relevant to the review questions, the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) was used by the two reviewers to independently appraise the quality of each study across two universal screening questions and five core criteria that are specific to…”
BackgroundMethodVicary, E., Kapadia, D., Bee, P., Bennion, M., & Brooks, H. L. (2024) · Journal of Mental Health · doi:10.1080/09638237.2024.2408237 · Cited by 69
“The MMAT quality appraisal tool (Hong et al., 2018) was used to assess the methodological quality of evidence.”
Nimbley, E., Sharpe, H., Maloney, E., Gillespie‐Smith, K., Tchanturia, K., & Duffy, F. (2024) · International Journal of Eating Disorders · doi:10.1002/eat.24311 · Cited by 40
Fijačko, N., Gosak, L., Debeljak, N., Skok, P., Štiglic, G., & Cilar, L. (2020) · Slovenian Nursing Review · doi:10.14528/snr.2020.54.2.2991 · Cited by 41
Infant feeding practices and later parent-reported feeding difficulties: a systematic review
Bąbik, K., Patro-Gołąb, B., Zalewski, B., et al. (2020) · Nutrition Reviews · doi:10.1093/nutrit/nuaa135 · Cited by 32
Ramage, E. R., Fini, N. A., Lynch, E. A., Marsden, D. L., Patterson, A. J., Said, C. M., & English, C. (2021) · Physical Therapy · doi:10.1093/ptj/pzab073 · Cited by 34
Alshammari, H., Al-Saeed, E., Ahmed, Z., & Aslanpour, Z. (2021) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s303101 · Cited by 48
“The first version of the MMAT in 2006 was piloted and went through interrater reliability testing.(18) It was revised in 2011 after being piloted in workshops, which led to the addition of new criteria to assess nonrandomised studies.”
MethodRuiz-Cosignani, D., Chen, Y., Cheung, G., Lawrence, M., Lyndon, M. P., Ma’u, E., & Ramalho, R. (2022) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x211069664 · Cited by 37
Family Adjustment to Hereditary Cancer Syndromes: A Systematic Review
Gomes, P. V. T., Pietrabissa, G., Silva, E., Silva, J., Matos, P. M., Costa, M. E., Bertuzzi, V., Silva, E., Neves, M. C., & Sales, C. M. D. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19031603 · Cited by 36
A systematic review of library makerspaces research
Kim, S. H., Jung, Y. J., & Choi, G. W. (2022) · Library & Information Science Research · doi:10.1016/j.lisr.2022.101202 · Cited by 39
“The analysis method was determined based on the categorization of research methods in MMAT.”
InfluentialPhysical health comorbidities in older adults with bipolar disorder: A systematic review
Warner, A., Holland, C., Lobban, F. A., Tyler, E., Harvey, D., Newens, C., & Palmier‐Claus, J. E. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.01.083 · Cited by 37
McIntyre, N., Crilly, J., & Elder, E. (2024) · International Emergency Nursing · doi:10.1016/j.ienj.2024.101437 · Cited by 31
Gonçalves, V., Ferreira, P. L., Saleh, M., Tamargo, C. L., & Quinn, G. P. (2022) · The Oncologist · doi:10.1093/oncolo/oyab051 · Cited by 33
Brisbine, B. R., Radcliffe, C. R., Jones, M. L. H., Stirling, L., & Coltman, C. E. (2022) · PLoS ONE · doi:10.1371/journal.pone.0278174 · Cited by 54
A Systematic Review of Barriers and Facilitators of Pain Management in Persons with Dementia
Liao, Y. K., Jao, Y., Berish, D. E., Hin, A. S., Wangi, K. Y. W., Kitko, L. A., Mogle, J. A., & Boltz, M. P. (2023) · Journal of Pain · doi:10.1016/j.jpain.2022.12.014 · Cited by 40
Shin, J., & Konlan, K. D. (2023) · Nursing Open · doi:10.1002/nop2.1613 · Cited by 41
“A systematic review of the determinants of medication adherence was conducted after an evaluation of the selected studies using the Mixed Method Appraisal tool (MMAT; Hong et al., 2018).”
BackgroundMethodInfluentialWhat Enables Child Sexual Abuse in Sport? A Systematic Review
Dodd, K., Solomon, C., Naughton, M. R., Salmon, P. M., & McLean, S. G. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231190666 · Cited by 39
“The MMAT appraisal criteria deemed three of the articles as not meeting the requirements for further screening, although these articles met the systematic literature review inclusion criteria and were included.”
MethodInfluentialSoeiro, R. E., Guida, J. P. D. S., da‐Costa‐Santos, J., & Costa, M. L. (2023) · Reproductive Health · doi:10.1186/s12978-023-01715-8 · Cited by 47
“[15, 17].”
MethodInfluentialWallace, E. R., O’Neill, S. M., & Lagdon, S. (2024) · Journal of Adolescence · doi:10.1002/jad.12308 · Cited by 46
Wang, G., Chang, F., Gu, Z., Kasraian, D., & van Wesemael, P. J. V. (2024) · BMC Public Health · doi:10.1186/s12889-024-18195-5 · Cited by 44
“The Mixed Methods Appraisal Tool (MMAT) was used independently by two authors (FC and GW) to assess the quality of the research [25].”
MethodSociotechnical analysis of factors influencing IoT adoption in healthcare: A systematic review
Kronlid, C., Brantnell, A., Elf, M., Borg, J., & Palm, K. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102675 · Cited by 46
Bhuiyan, M. A., Galdes, N., Cuschieri, S., & Hu, P. (2024) · Journal of Health Population and Nutrition · doi:10.1186/s41043-024-00607-2 · Cited by 47
“The assessment of potential biases in studies chosen for this review was conducted using the updated Mixed Methods Appraisal Tool (MMAT), described by [42].”
MethodCho, H., Oh, O., Greene, N., Gordon, L., Morgan, S., Walke, L. M., & Demiris, G. (2025) · The Journals of Gerontology Series A · doi:10.1093/gerona/glaf024 · Cited by 41
Zhou, Y. J., Zhang, A., van Klinken, R. D., Jones, D., & Wang, J. (2025) · npj Science of Food · doi:10.1038/s41538-025-00381-3 · Cited by 45
“The quality of all included articles was assessed using “ The Mixed Methods Appraisal Tool (MMAT) version 2018 ” 85 .”
Shahin, W., Kennedy, G. A., & Stupans, I. (2020) · Pharmacy · doi:10.3390/pharmacy8030147 · Cited by 36
“Each eligible study included in this review was assessed using the Mixed Methods Appraisal Tool (MMAT) version 2018 [21], which comprises different methodological criteria depending on the type of the study.”
MethodPathways to care in at‐risk mental states: A systematic review
Allan, S. M., Hodgekins, J., Beazley, P. I., & Oduola, S. (2020) · Early Intervention in Psychiatry · doi:10.1111/eip.13053 · Cited by 41
“Methodological quality of the studies was measured using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodInternet Use and Resilience in Adolescents: A Systematic Review
Sage, M., Randolph, K. A., Fitch, D., & Sage, T. (2020) · Research on Social Work Practice · doi:10.1177/1049731520967409 · Cited by 32
Surgery cancellation: A scoping review of patients' experiences
Viftrup, A., Dreyer, P., Nikolajsen, L., & Holm, A. (2021) · Journal of Clinical Nursing · doi:10.1111/jocn.15582 · Cited by 31
Osokpo, O. H., James, R., & Riegel, B. J. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14804 · Cited by 42
Women in Relationships and Their Pornography Use: A Systematic Review and Thematic Synthesis
Litsou, K., Graham, C. A., & Ingham, R. (2021) · Journal of Sex & Marital Therapy · doi:10.1080/0092623x.2021.1885532 · Cited by 44
Use of the pre-medical emergency team tier of rapid response systems: A scoping Review
Sprogis, S. K., Currey, J., Jones, D. A., & Considine, J. (2021) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2021.103041 · Cited by 36
Hospital contextual factors affecting the implementation of health technologies: a systematic review
Grossi, A., Hoxhaj, I., Gabutti, I., Specchia, M. L., Cicchetti, A., Boccia, S., & de Waure, C. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06423-2 · Cited by 45
“Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers.”
MethodInfluentialChua, J. Y. X., Choolani, M. A., Chee, C. Y. I., Chan, Y. H., Lalor, J., Chong, Y. S., & Shorey, S. (2023) · Journal of Midwifery & Women s Health · doi:10.1111/jmwh.13472 · Cited by 43
Park, J., Kang, S., Seok, D., Baek, Y. J., An, S. Y., Lee, J.-A., Jun, A., & Kim, S. (2023) · Malaria Journal · doi:10.1186/s12936-023-04452-2 · Cited by 37
“The MMAT is a methodological quality appraisal tool designed for systematic reviews encompassing studies with various types of study designs [14].”
MethodTing, R. S., Lewis, D. P., Yang, K. X., Nguyen, T. A., Sarrami, P., Daniel, L., Hourigan, S. T., King, K. L., Lassen, C., Sarrami, M., et al. (2023) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000003923 · Cited by 31
“Quality Appraisal Included studies were appraised using a modified Oxford Center for Evidence-based Medicine rating and Mixed Methods Appraisal Tool.(16,17)”
MethodMcWilliams, L., Paisi, M., Middleton, S., Shawe, J., Thornton, A., Larkin, M., Taylor, J., & Currie, J. (2022) · Journal of Advanced Nursing · doi:10.1111/jan.15387 · Cited by 42
“The studies were appraised using the Mixed Methods Appraisal Tool (MMAT) Version 18 (Hong et al., 2018 ).”
MethodSchwarz, A. F., Winkens, L. H. H., de Vet, E., Ossendrijver, D., Bouwsema, K., & Simons, M. (2022) · JMIR mhealth and uhealth · doi:10.2196/40898 · Cited by 45
“Study quality was assessed according to the Mixed Method Assessment Tool, and one-third of all screening and data extraction were double coded by a second reviewer.”
MethodInfluentialOketch, S. Y., Ochomo, E. O., Orwa, J., Mayieka, L., & Abdullahi, L. H. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-067164 · Cited by 46
Bostan, S., Johnson, O. A., Jaspersen, L. J., & Randell, R. (2024) · Journal of Medical Internet Research · doi:10.2196/45242 · Cited by 69
Planetary boundaries transgressions: A review on the implications to public health
Kemarau, R. A., Sakawi, Z., Eboy, O. V., Suab, S. A., Ibrahim, M. F., Rosli, N. N., & Nor, N. N. F. M. (2024) · Environmental Research · doi:10.1016/j.envres.2024.119668 · Cited by 38
What’s going on with teleworking? a scoping review of its effects on well-being
Vacchiano, M., Fernandez, G., & Schmutz, R. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305567 · Cited by 59
Trainum, K., Liu, J., Hauser, E., & Xie, B. (2024) · JMIR Aging · doi:10.2196/58629 · Cited by 41
Khan, T., Malik, S., Rafeekh, L., Halder, S., Desai, S., & Bhattacharya, S. D. (2024) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2024.2411823 · Cited by 43
A systematic review on risk factors and reasons for e-cigarette use in adolescents
Villanueva‐Blasco, V. J., Ferri, L. B., & Vázquez‐Martínez, A. (2025) · Tobacco Induced Diseases · doi:10.18332/tid/196679 · Cited by 61
Barriers for access and utilization of dementia care services in Europe: a systematic review
Sorrentino, M., Fiorilla, C., Mercogliano, M., Stilo, I., Esposito, F., Moccia, M., Lavorgna, L., Salvatore, E., Sormani, M. P., Majeed, A., et al. (2025) · BMC Geriatrics · doi:10.1186/s12877-025-05805-z · Cited by 46
“The quality of the included papers was evaluated using the Mixed Methods Appraisal Tool (MMAT), revised version [35].”
A typology of systematic reviews for synthesising evidence on health care
MacEntee, M. I. (2019) · Gerodontology · doi:10.1111/ger.12439 · Cited by 32
Online health communities, contributions to caregivers and resilience of older adults
Kamalpour, M., Aghdam, A. R., Watson, J., Tariq, A., Buys, L. R., Eden, R., & Syed, R. (2020) · Health & Social Care in the Community · doi:10.1111/hsc.13247 · Cited by 34
Sørensen, K. M. (2021) · Library & Information Science Research · doi:10.1016/j.lisr.2020.101067 · Cited by 30
Computerized Decision Support Systems for Nursing Homes: A Scoping Review
Abdellatif, A., Bouaud, J., Lafuente‐Lafuente, C., Belmin, J., & Séroussi, B. (2021) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2021.01.080 · Cited by 38
Cassarino, M., Shahab, S., & Biscaya, S. (2021) · Sustainability · doi:10.3390/su13148086 · Cited by 37
“We employed the mixed methods appraisal tool (MMAT) [39] to assess the quality of the studies included in the review (see Supplementary File S4).”
MethodMcCann, E., Brown, M. J., Martin, C. J. H., Murray, K. B., & McCormick, F. (2021) · Midwifery · doi:10.1016/j.midw.2021.103102 · Cited by 35
“To reduce the potential of bias, two authors individually quality assessed all papers presented in Table 2 using the mixed methods appraisal tool (MMAT) (Hong et al., 2018).”
MethodInfluentialPatel, J., Tolppa, T., Biccard, B. M., Fazzini, B., Haniffa, R., Marletta, D., Moonesinghe, R., Pearse, R. M., Sutharshan, V., Stephens, T., et al. (2022) · World Journal of Surgery · doi:10.1007/s00268-022-06621-x · Cited by 31
“Quality assessments were carried out independently by two researchers using the Mixed Methods Appraisal Tool (MMAT) for descriptive statistics of included studies [18].”
Chua, C. M. S., Mathews, J., Ong, M. S. B., Liew, K. K., & Shorey, S. (2023) · Women and Birth · doi:10.1016/j.wombi.2022.06.011 · Cited by 28
Bujold, M., Pluye, P., Légaré, F., the Participatory Review Team, Hong, Q. N., Beaulieu, M., Bush, P. L., Couturier, Y., Sherif, R. E., Gagnon, J., et al. (2022) · BMC Family Practice · doi:10.1186/s12875-022-01879-5 · Cited by 44
“Critical appraisal of included studies We used the Mixed Methods Appraisal Tool (MMAT) to appraise and describe the methodological quality of the included qualitative, quantitative and mixed methods studies [26].”
MethodSelf-citationPrevalence of self-medication in Ghana: a systematic review and meta-analysis
Opoku, R., Dwumfour-Asare, B., Agrey-Bluwey, L., Appiah, N. E., Ackah, M., Acquah, F., Asenso, P. F., & Issaka, A.-A. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-064627 · Cited by 30
Interdisciplinary Mixed Methods Systematic Reviews: Theoretical and Practical Considerations
Nowell, L. S., Paolucci, A., Dhingra, S., Jacobsen, M., Lorenzetti, D., Lorenzetti, L., & Paolucci, E. O. (2022) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2022.100295 · Cited by 37
Kelly, P., Saab, M. M., Hurley, E. J., Heffernan, M., Goodwin, J., Mulud, Z. A., O’Malley, M., O’Mahony, J., Curtin, M., Groen, G., et al. (2023) · Journal of Child & Adolescent Trauma · doi:10.1007/s40653-023-00524-2 · Cited by 47
“The MMAT assists in assessing the quality of five study categories: qualitative studies, randomised controlled trials (RCTs), non-randomised studies, quantitative descriptive studies, and mixed methods studies.”
MethodInfluentialReducing unplanned hospital admissions from care homes: a systematic review
Chambers, D., Cantrell, A. J., Preston, L., Marincowitz, C., Wright, L., Conroy, S., & Gordon, A. L. (2023) · Health and Social Care Delivery Research · doi:10.3310/klpw6338 · Cited by 38
Nutritional counseling in athletes: a systematic review
Fiorini, S., Neri, L. D. C. L., Guglielmetti, M., Pedrolini, E., Tagliabue, A., Quatromoni, P. A., & Ferraris, C. (2023) · Frontiers in Nutrition · doi:10.3389/fnut.2023.1250567 · Cited by 51
Participatory research with carers: A systematic review and narrative synthesis
Bowness, B., Henderson, C., Akhter‐Khan, S. C., Akiba, M., & Lawrence, V. C. (2023) · Health Expectations · doi:10.1111/hex.13940 · Cited by 35
“9 MMAT, Mixed Methods Appraisal Tool. unique experiences , and carers create change (see Supporting Information S5).”
BackgroundMethodInfluentialThe impact of stigma on the management of type 1 diabetes: A systematic review
Embick, R., Jackson, M. C., & Stewart, R. (2024) · Diabetic Medicine · doi:10.1111/dme.15299 · Cited by 32
Gender, sex and complex PTSD clinical presentation: a systematic review
Lonnen, E., & Paskell, R. (2024) · European Journal of Psychotraumatology · doi:10.1080/20008066.2024.2320994 · Cited by 37
“Suitable for qualitative, quantitative and mixed methods studies, the MMAT assesses quality against five criteria relevant to each of five study designs.”
MethodBolster‐Foucault, C., Vedel, I., Busa, G., Hacker, G., Sourial, N., & Quesnel‐Vallée, A. (2024) · Age and Ageing · doi:10.1093/ageing/afae166 · Cited by 43
Self-citationThe reality of the gamification methodology in Primary Education: A systematic review
Rodríguez, J. M. R., Menéndez, A. M., Alonso-García, S., & Maldonado, J. J. V. (2024) · International Journal of Educational Research · doi:10.1016/j.ijer.2024.102481 · Cited by 93
Sousa, M., Machado, A. B., Pinheiro, M. L., Pereira, B., Caridade, S., Almeida, T. C., Cruz, A. R., & Cunha, O. S. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251320978 · Cited by 41
“The methodological quality of all included studies was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Buckley, L., Hall, L. M., Price, S. L., Visekruna, S., & McTavish, C. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-096333 · Cited by 52
Li, K., Wang, P., & Chen, G. (2025) · Teaching and Teacher Education · doi:10.1016/j.tate.2025.105219 · Cited by 69
The Impact of Reassurance on Musculoskeletal (MSK) Pain: A Qualitative Review
Cheung, L. K., & Soundy, A. (2021) · Behavioral Sciences · doi:10.3390/bs11110150 · Cited by 37
“The mixed methods appraisal tool (MMAT) [21] was used to assess the quality of the mixed methods designs.”
MethodInfluentialGroup-Based Interventions for Carers of People With Dementia: A Systematic Review
McLoughlin, B. (2022) · Innovation in Aging · doi:10.1093/geroni/igac011 · Cited by 41
“The Joanna Briggs Institute (JBI) (2016) Critical Appraisal checklist for quasi-experimental studies were used for nonrandomised quantitative studies and the Mixed-Methods Appraisal Tool (MMAT) checklist was used to evaluate mixed methods research (Hong et al., 2018).”
BackgroundMethodHow to support fathers of preterm infants in early parenthood – An integrative review
Holm, K. G., Aagaard, H., Maastrup, R., Weis, J., Feenstra, M. M., Haslund‐Thomsen, H., Hägi‐Pedersen, M., Nilsson, I. M. S., Kristensen, I. H., Dreier, S., et al. (2022) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2022.10.001 · Cited by 38
“Further, the included studies were assessed using the Mixed Methods Appraisal Tool recommended for reviews, including original qualitative and quantitative studies (Hong, 2018).”
MethodHanlon, C. A., McIlroy, D., Poole, H. M., Chopra, J., & Saini, P. (2022) · Health Expectations · doi:10.1111/hex.13661 · Cited by 36
Shpendi, S., Norman, P., Gibson-Miller, J., & Webster, R. (2025) · Women's Health · doi:10.1177/17455057251324309 · Cited by 29
Dimla, B., Parkinson, L., Wood, D., & Powell, Z. (2022) · Australian Journal on Ageing · doi:10.1111/ajag.13138 · Cited by 29
AlSwayied, G., Guo, H., Rookes, T., Frost, R., & Hamilton, F. L. (2022) · JMIR mhealth and uhealth · doi:10.2196/40271 · Cited by 42
“Two authors (GS and HG) assessed the methodological quality of each included study independently using the Mixed Methods Appraisal Tool [67] and discussed their assessments to achieve consensus.”
MethodMobile Health Self-management Support for Spinal Cord Injury: Systematic Literature Review
Bernard, R. M., Seijas, V., Davis, M., Volkova, A., Diviani, N., Lüscher, J., & Sabariego, C. (2023) · JMIR mhealth and uhealth · doi:10.2196/42679 · Cited by 31
“…tool for assessing risk of bias in randomized trials [28] Mixed methods studies • Mixed Methods Appraisal Tool for systematic mixed studies reviews [29] Other observational studies • Joanna Briggs Institute Checklist for Analytical Cross-Sectional Studies [30] Qualitative studies • Joanna Briggs…”
Neylon, K., Condren, C., Guérin, S., & Looney, K. (2023) · Journal of Adolescent and Young Adult Oncology · doi:10.1089/jayao.2022.0130 · Cited by 35
Usability and Acceptability of Clinical Dashboards in Aged Care: Systematic Review
Siette, J., Dodds, L., Sharifi, F., Nguyen, A., Baysari, M. T., Seaman, K. L., Raban, M. Z., Wabe, N., & Westbrook, J. I. (2023) · JMIR Aging · doi:10.2196/42274 · Cited by 35
“Each study category has 5 assessment criteria, which are scored as either “yes—criterion met,” “no—criterion not met,” or “unclear/can’t tell whether criterion met” [29].”
MethodInfluentialCole, L., Ridings, L. E., & Phillips, S. M. (2023) · Pediatric Cardiology · doi:10.1007/s00246-023-03227-5 · Cited by 29
“The Mixed Method Appraisal Tool (MMAT) [20] was used to evaluate the quality of study design, participant recruitment, data collection and analysis, and presentation of findings.”
Midje, H. H., Nyborg, V. N., Nordsteien, A., Øvergård, K. I., Brembo, E. A., & Torp, S. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15804 · Cited by 43
Williams, A., Lennox, L., Harris, M. J., & Antonacci, G. (2023) · Implementation Science · doi:10.1186/s13012-023-01311-1 · Cited by 40
“The Mixed Methods Appraisal Tool (MMAT), version 2018 [64] was used for the quality assessment of each article.”
MethodMental health services implementation in Colombia–A systematic review
Garavito, G. A. A., Burgess, R. A., Sanguineti, M. C. D., Peters, L. E. R., & Juan, N. V. S. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001565 · Cited by 37
“This tool was selected because it is feasible to evaluate mixed studies reviews [72].”
MethodNaughton, M. R., Salmon, P. M., Kerhervé, H. A., & McLean, S. G. (2024) · Journal of Sports Sciences · doi:10.1080/02640414.2024.2306056 · Cited by 43
“The MMAT was developed to assess the quality studies that use different study designs (qualitative, quantitative, and mixed methods) in a single tool (Hong et al., 2018).”
FinTech application in Islamic social finance in Asia region: a systematic literature review
Kasmon, B., Ibrahim, S. S., Daud, D. H., Hisham, R. R. I. R., & Prajanti, S. D. W. (2024) · International Journal of Ethics and Systems · doi:10.1108/ijoes-07-2023-0155 · Cited by 59
Tang, P., Kostyrka‐Allchorne, K., Butura, A.-M., Phillips, J., & Sonuga-Barke, E. J. S. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102465 · Cited by 41
Ng, W. H. D., Ang, W. H. D., Fukahori, H., Goh, Y. S., Lim, W. S., Siah, C. J. R., Seah, B., & Liaw, S. Y. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17375 · Cited by 45
“The MMAT assesses the quality of different study designs, and it includes five key criteria to efficiently and critically appraise each type of study design, as well as two screening questions for all study designs (Hong et al., 2018).”
Sohn, J.-S., Lee, E., Kim, J., Oh, H.-K., & Kim, E. J. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1628216 · Cited by 49
“To appraise the methodological quality of the included studies, we use the mixed methods appraisal tool (MMAT), version 2018 (32).”
Kanda, K., Blythe, S. L., Grace, R., & Kemp, L. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-07666-3 · Cited by 45
“However, no studies were excluded on the basis of the overall quality scores as it is discouraged by Hong and colleagues [46], and also in order to have a comprehensive summary of the existing evidence on parent satisfaction with home visiting care and interventions.”
BackgroundMethod
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.