The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 3,101–3,200 of 4,578
Secondary school students’ creativity in STEM education: A systematic literature review
Guridi-Bikuña, A., Bilbao, A., & Álvarez‐Huerta, P. (2026) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2026.102264 · Cited by 2
Guerrero-Calero, J. M., Ganchozo, B. S. I., Garcia-Rodriguez, R., & Moran-González, M. R. (2026) · Revista Científica Zambos. · doi:10.69484/rcz/v5/n2/172 · Cited by 1
“…aplicando la herramienta CASP para estudios cualitativos y el MMAT 2018 en el caso de estudios mixtos o revisiones con evidencia heterogénea, lo que permitió ponderar la solidez de los resultados y los posibles riesgos de sesgo en la interpretación final (Hong et al., 2018; Pollock et al., 2023).”
Almkhlafi, T. S. S., Brazil, K., & Porter, S. (2026) · Nurse Education in Practice · doi:10.1016/j.nepr.2026.104879 · Cited by 1
Applied sport psychology in esports: A scoping review of intervention strategies
Leis, O., Shin, J., Welsh, M. R., Wunsch, L. S. L., & Sharpe, B. T. (2026) · International Journal of Sports Science & Coaching · doi:10.1177/17479541261455245 · Cited by 5
Lue, L., Reis, K., Kanasa, H., Rutherford, S., & Desha, C. (2026) · Environmental Education Research · doi:10.1080/13504622.2026.2675389 · Cited by 2
Artificial intelligence for engaging neurodiverse learners in K-12 education: a systematic review
Uddın, M. M., Luo, F., & Moon, J. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2679063 · Cited by 2
Ismail, N., Ayub, A. F. M., & Zhang, T. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2673624 · Cited by 1
Youlyouz, A., Chat, M., Ouaali, A. A., Boumediane, M. B., Azzi, O., Elhousni, Z., Janati‐Idrissi, R., Boubih, S., & liemlahi, H. (2026) · Multidisciplinary Reviews · doi:10.31893/multirev.2026561 · Cited by 2
“The methodological quality of included empirical studies was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018), a critical appraisal tool designed for systematic reviews encompassing qualitative, quantitative, and mixed-methods studies.”
Krisnanda, V. D., Setyosarı, P., Ramli, M., Setiyowati, A. J., & Aminah, S. (2026) · Multidisciplinary Reviews · doi:10.31893/multirev.2026558 · Cited by 1
“The methodological quality of the included studies was assessed via the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2018).”
Jarrah, S. A. (2026) · Journal of International Logistics and Trade · doi:10.1108/jilt-01-2026-0014 · Cited by 1
Fenech, G., Fondacaro, D. V., & New, K. J. (2026) · Journal of Rare Diseases · doi:10.1007/s44162-026-00207-0 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT; [25] was used to assess the quality of the chosen manuscripts (Table 2).”
He, Y., Long, P., Chen, R., Zhi, D., Chen, Y., Ma, L., Calhoun, V. D., Qin, S., He, Y., Dong, Q., et al. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10189-6 · Cited by 2
“Consistent with MMAT guid - ance, we did not compute a single numerical quality score, because summary scores can obscure domain-specific strengths and weaknesses.”
Fahey, J. T., Robles-Palazón, F. J., Comfort, P., & Ripley, N. (2026) · PLoS ONE · doi:10.1371/journal.pone.0351121 · Cited by 3
“The methodological quality of articles was assessed (JF & FJRP) using a mixed methods appraisal tool (MMAT) [93] with any conflicts resolved by NR, and the findings provided for descriptive purposes only.”
Salim, N. S. M., Idris, I. B., Nor, N. K., Roslan, S., Rashid, N. A., Ibrahim, R. B., & Rahman, A. F. A. (2026) · PLoS ONE · doi:10.1371/journal.pone.0351969 · Cited by 1
“The Mixed Methods Appraisal Tool (MMAT, 2018) was used to evaluate the study quality [30].”
In-Depth Breastfeeding Training for Healthcare Professionals: A Systematic Review
Lopez-Bassols, I., Cameron, J. E., Dalcin, C. B., & McFadden, A. (2026) · Journal of Human Lactation · doi:10.1177/08903344261446138 · Cited by 1
Nurses’ Experiences Using AI in Clinical Practice: Systematic Review
Scott, A. J. S., Zhao, Q., Pan, J., Brown, B. C., & Dowding, D. (2026) · JMIR Nursing · doi:10.2196/91238 · Cited by 6
“A key strength of this review is its comprehensive synthesis across diverse AI applications and settings, supported by a transparent search strategy and appraisal process using the MMAT [22].”
Pai, K. S. R., Shetye, S., Patil, R. V., Acharya, S., Kulal, K. S., Patil, A., & Kamath, S. U. (2026) · Oxidative Medicine and Cellular Longevity · doi:10.1155/omcl/8685788 · Cited by 1
“For observational, clinical, and mixed-method studies, we employed the Mixed Methods Appraisal Tool (MMAT) [76], which facilitates a systematic evaluation of qualitative, quantitative, and mixed-method research designs.”
Benafif, S., EROĞLU, H. S., Conroy, P., et al. (2026) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2026.2693243 · Cited by 1
Polatuly, S., Ramankulov, S. J., Yedilbayev, Y., et al. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1812251 · Cited by 1
Munasinghe, N. L., Fiolet, R., & Hutchinson, A. M. (2026) · BMC Emergency Medicine · doi:10.1186/s12873-026-01660-2
“The methodological quality of included studies was appraised using the Mixed Methods Appraisal Tool (MMAT) version 2018 [16].”
TOKGÖZ, Ö. (2026) · Uluslararası Eğitim Programları ve Öğretim Çalışmaları Dergisi · doi:10.31704/ijocis.1918818 · Cited by 1
Humanistic Qualities in Physical Education: A Systematic Review and Meta-Synthesis
Dong, W., Huang, P., Xiang, C., Wang, X., & Jiao, Y. (2026) · Quest · doi:10.1080/00336297.2026.2695366 · Cited by 1
Ortiz-Terrinches, J.-F., Sánchez-Cabrero, R., & Mena, M. S. (2026) · Administrative Sciences · doi:10.3390/admsci16070318 · Cited by 5
Singh, R., Anand, S., Johry, A., & Kar, G. (2026) · HERD Health Environments Research & Design Journal · doi:10.1177/19375867261451751 · Cited by 2
“All the selected articles were organized as per their study type and relative quality using the MMAT and Levels of Evidence for Healthcare Design in Figure 2.”
InfluentialEffectiveness and cost-effectiveness of general health checks in adults: A systematic review
Wang, Y., Zhao, Q., Zhu, X., Wang, L., Xiang, L., Liu, Q., Yang, T., & Liu, J. (2026) · BMC Public Health · doi:10.1186/s12889-026-28436-4 · Cited by 1
Dimensions and assessments of advertising literacy: A systematic review
Yao, E. Z., & List, A. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100817 · Cited by 1
Irwansyah, I. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.103246 · Cited by 2
Jia, Z., Dai, D. W., Aryadoust, V., & Hua, Z. (2026) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2026.2694603 · Cited by 1
Maruf, I. R., Noor, N. S., Omar, F. M. S., & Mubaroq, S. R. (2026) · Frontiers in Political Science · doi:10.3389/fpos.2026.1817342 · Cited by 1
An, S., Song, Y., & Jiang, B. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10201-z · Cited by 3
“The Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) was applied because it is suitable for reviews including qualitative, quantitative, and mixed methods studies.”
Özturan, İ. U., Daylan, P., Alyeşil, C., Kitapçıoğlu, D., & Altıntaş, L. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1873126 · Cited by 3
“Methodological quality was appraised using the Mixed Methods Appraisal Tool (MMAT) version 2018 (19).”
Kirsman, N., Refaeli, L. B., Enav, Y., & Mayer, Y. (2026) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-026-00580-8 · Cited by 1
Saad, N., Wu, J., Hill, M., Dorris, C. S., Liu, J., Wernet, F., Wu, Y.-H., Anand, J. K., Sawez, A., & Yu, W. (2026) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0013758 · Cited by 1
MURALEEDHARAN, D., & Vysakh, C. (2026) · The International Information & Library Review · doi:10.1080/10572317.2026.2704164 · Cited by 1
Mamabolo, E., Mogale, T. E., Mugari, E., Mabitsela, K. E., & Ayisi, K. K. (2026) · Frontiers in Sustainable Food Systems · doi:10.3389/fsufs.2026.1858469 · Cited by 1
Yap, W. S., Cheong, W. L., & Yeap, L. (2026) · Journal of Pharmaceutical Policy and Practice · doi:10.1080/20523211.2026.2701922 · Cited by 2
“Due to the heterogeneity of study designs, methodological quality of the empirical publications was assessed using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018), with methodological quality rated as high (all five criteria met), moderate (at least four criteria met), or low (less than four criteria met).”
InfluentialYuan, Y., Deng, M., Chen, H., Lu, X., Chen, X., Bai, D., Gao, H., Hou, C., & Gao, J. (2026) · Therapeutic Advances in Endocrinology and Metabolism · doi:10.1177/20420188261469945 · Cited by 2
Marfo, R., Miezah, D., Abubakari, M., Assuah, E., Lartey, S. D., Wilson, D., Asamoah, E., Kyei, E. F., Ansong, R., & Boakye, M. D. S. (2026) · Diabetic Medicine · doi:10.1111/dme.70414 · Cited by 2
Xia, Y., Wang, T., & Ang, Y. Q. (2026) · Sustainable Cities and Society · doi:10.1016/j.scs.2026.107774 · Cited by 1
Paredes, R. R., & Bernal, D. P. (2026) · TRASCENDERE · doi:10.56219/trascendere.v3i7.5979 · Cited by 1
“Se presenta una evaluación cualitativa detallada sin cálculo de puntajes numéricos, siguiendo las recomendaciones de los autores de la herramienta (Hong et al., 2018).”
Chaux, A. (2026) · Exploration of Targeted Anti-tumor Therapy · doi:10.37349/etat.2026.1002397 · Cited by 1
Tandung, S. S., Obel, O., Zulaikha, I. '., Ikbar, R. D., Jailani, M. I., Priandi, A. C., Alemoka, C. K., Sabar, F. C., Taneo, R. F. S., Herawati, A. (. T., et al. (2026) · F1000Research · doi:10.12688/f1000research.189247.1
Soares, G. V. D., Funan, S. E., Adhania, D., Nupus, D. H., Bria, P. M., Kamisopa, J. E., Everest, M. E., Sarda, S., Novianto, H. A., Chilavert, W. P. C., et al. (2026) · F1000Research · doi:10.12688/f1000research.189349.1
Rethinking urban green space vitality: A systematic review and future research agenda
Jiang, S., Cheng, K., Wang, C., et al. (2026) · Urban forestry & urban greening · doi:10.1016/j.ufug.2026.129641 · Cited by 1
Baydeniz, E. (2026) · International Journal of Tourism Cities · doi:10.1080/20565607.2026.2721576 · Cited by 1
Generative Large Language Models in Mental Health Care Settings: Systematic Review and Meta-Analysis
Leung, J., Johnson, B. H., Mcrae, K., Fong, S., Gonzalez, P. C., McClure‐Thomas, C., Sun, T., Kern, N., Wong, Y. M., Liu, R., et al. (2026) · JMIR AI · doi:10.2196/87730 · Cited by 4
“Methodological quality and risk of bias were appraised using the Mixed Methods Appraisal Tool (MMAT), which was selected because it provides design-specific criteria applicable to different study designs [24].”
Lund, J. C., Kongensgaard, F. T., Brøchner, A. C., & Nedergaard, H. K. (2026) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.70334 · Cited by 1
Dix‐Cooper, L., Dawes, M. G., & Park, M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.027 · Cited by 1
Aylward, S. M., Farrell, A., Walsh, A., Godwin, M. S., Chafe, R., & Asghari, S. (2021) · JMIR Research Protocols · doi:10.2196/28196 · Cited by 1
“Included studies will be appraised by using the 2018 version of the Mixed Methods Appraisal Tool [19]—a reliable tool that can be applied to the appraisal of quantitative, qualitative, and mixed methods study designs [19-21].”
MethodA Systematic Review on the Impact of COVID-19 to Communities
Sakip, S. R. M., Rahim, P. R. M. A., & Salleh, M. N. M. (2022) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v12-i4/12866 · Cited by 1
Lukacs, M. J., Kowalski, K. L., Peters, N., Stanley, M., & Rushton, A. B. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061475 · Cited by 2
Habilidade de manejo de famílias de crianças com condições crônicas para o cuidado no domicílio
Felizardo, M. J. D. A., da Silva, J. B., Neves, E. T., et al. (2022) · Escola Anna Nery · doi:10.1590/2177-9465-ean-2022-0071pt · Cited by 1
Nursing assessment and management of nutrition in older people with cancer: An integrative review
Nasrawi, D., Reid, C. J., Lee, M. F., & Coyne, E. (2022) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2022.06.012 · Cited by 3
A Systematic Review of Tropical Disease Prevalence among Migrants
Arifin, M. A., Peng, C. S., Baharudin, U. M., Baharudin, M. H., Rahim, M. A. A., Rahim, S. S. S. A., Atil, A., Madrim, M. F., Ahmad, Z. N. B. S., Mokti, K., et al. (2022) · Open Access Macedonian Journal of Medical Sciences · doi:10.3889/oamjms.2022.10211 · Cited by 3
“Next, we appraised the selected articles’ quality using the mixed methods appraisal tool (MMAT) Version 2018 [7] to corroborate the articles’ quality for reviewing.”
Guidelines for conducting rapid reviews in psychology research
du Toit, R. E., & de Klerk, W. (2022) · Journal of Psychology in Africa · doi:10.1080/14330237.2022.2121468 · Cited by 3
“For mixed-method studies, the quality assessment tool suggested includes the Mixed-method Appraisal Tool (MMAT: Hong et al., 2018).”
MethodImplementing a discharge process for patients undergoing elective surgery: Rapid review
Teixeira, M. J. C., Khouri, M., Martinez, E., & Bench, S. D. (2023) · International Journal of Orthopaedic and Trauma Nursing · doi:10.1016/j.ijotn.2023.101001 · Cited by 5
Sarıkaya, E. E., Alkan, M. F., Cladera, M., Leavy, A., Sevim, Ö. M., Paul, C. I., Primi, C., Schau, C., & Üstün, U. (2022) · Social Science Protocols · doi:10.7565/ssp.v5.6949 · Cited by 2
Sari, D. W., Aurizki, G. E., Indarwati, R., Farapti, F., Rekawati, E., & Takaoka, M. (2023) · JMIR Research Protocols · doi:10.2196/44201 · Cited by 2
“The stages will follow the preferred reporting items of the Mixed Methods Appraisal Tool [33,34].”
Ferreira, K. M., Balsanelli, A. P., & dos Santos, J. L. G. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6554.3935 · Cited by 1
“2 (5,2) 11 (28,9) 18 (47,3) 5 (13,1) 2 (5,2)”
MethodKivuti-Bitok, L., Cheptum, J. J., Mutwiri, M., Wanja, S., & Ngune, I. (2023) · African Journal of Health Nursing and Midwifery · doi:10.52589/ajhnm-mfh6yys8 · Cited by 2
Abujaber, A. A., & Nashwan, A. J. J. (2023) · Evidence-Based Nursing · doi:10.1136/ebnurs-2023-103707 · Cited by 1
de Campos, A. C., Hidalgo‐Robles, Á., Longo, E., Shrader, C., & Paleg, G. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15716 · Cited by 1
Kinch, M., Kroll, T., Stokes, D., & Somanadhan, S. (2023) · HRB Open Research · doi:10.12688/hrbopenres.13684.2 · Cited by 1
Hassabo, M., Mohamed, A. M. A., & Khaira, M. B. (2023) · Journal of Pharmaceutical Research International · doi:10.9734/jpri/2023/v35i237420 · Cited by 1
“Studies were classified as either low (scoring ≤3) or high (score >3) depending on whether participants answered: "yes" (1 point) or "no" (0 points) [32].”
Freitag, A. P., Schmetz, A., Alili, H. E., et al. (2023) · Medical Research Archives · doi:10.18103/mra.v11i8.4326 · Cited by 1
Factors Determining Turnaround Strategies for Declining, Large, High-tech organizations
Jeyasingh, B. B. F. (2023) · Journal of Business and Management Studies · doi:10.32996/jbms.2023.5.5.16x · Cited by 2
Centenaro, A. P. F. C., da Silva, R. M., Franco, G. P., et al. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6669.4003 · Cited by 1
Alaboud, N., Jani, Y. H., & Franklin, B. D. (2023) · International Journal of Pharmacy Practice · doi:10.1093/ijpp/riad074.003 · Cited by 1
Malani, S. N., Acharya, S., & Shukla, S. (2023) · Cureus · doi:10.7759/cureus.51030 · Cited by 3
Linking betrayal and mental contamination in OCD: A mixed-methods systematic narrative review
Nielsen, V. S., Bream, V., & Salkovskis, P. M. (2024) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2024.100862 · Cited by 3
Jeraldo, R. I. E., Zabihi, S., Miranda‐Castillo, C., Marshall, C. R., & Cooper, C. (2026) · BMJ Global Health · doi:10.1136/bmjgh-2024-017630 · Cited by 1
“SZ and RIEJ independently assessed quality of included studies using the RCT items from the Mixed Methods Assessment Tool (MMAT), 19 which score domain criteria: randomisation process, treatment assignment, follow-up, data collection methods and outcome measurement; we included an additional item…”
Gormley, O., Cunningham, T., McFadden, S., & Toner, P. (2024) · Journal of Forensic Psychology Research and Practice · doi:10.1080/24732850.2024.2328256 · Cited by 3
“Following this, the completed MMAT forms were divided between TC and PT to review and any conflicting appraisals were discussed by the research team until a majority opinion was agreed.”
InfluentialAssessment of Standardized Care Plans for People with Chronic Diseases in Primary Care Settings
García, G. R., Cámara-Liebana, D., Soler, R. S., Pau-Perich, E., Sitjar-Suñer, M., Mantas‐Jiménez, S., Roqueta-Vall-llosera, M., & Malagón-Aguilera, M. D. C. (2024) · Nursing Reports · doi:10.3390/nursrep14020062 · Cited by 4
Carga cognitiva en el aprendizaje colaborativo: Una revisión sistemática
Dávila, L. A. O., Alburqueque, I. L. V., Ledesma-Pérez, F., & Amaya, W. H. C. (2024) · doi:10.31876/rcs.v30i2.41917 · Cited by 13
Ismail, N. H., & Osman, K. (2024) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v13-i2/21226 · Cited by 1
“Based on the Mixed Methods Assessment Tool (MMAT) developed by Hong et al (2018), this will facilitate the researchers to appraise mixed-methods studies within a systematic review.”
MethodSGLT-2 Inhibitors and Their Correlation With Kidney Stones. A Systematic Review
Alshehri, B. J., & Shalabi, H. (2024) · Journal of Health and Rehabilitation Research · doi:10.61919/jhrr.v4i2.743 · Cited by 2
A systematic review on carbon emission of light duty vehicles in urban environment
Singam, V. T., Zahari, H. M., & Rafiuddin, N. M. (2024) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2024.100924 · Cited by 1
Promoting and Sustaining Sexual Health as a Health Resource in Prisons - A Scoping Review
Solberg, A. S., Valde, E., Helgeland, K., et al. (2024) · Bergen Journal of Criminal Law & Criminal Justice · doi:10.15845/bjclcj.v12i1.4274 · Cited by 1
Reducing Recidivism through Entrepreneurship: A Systematic Literature Review
Zainal, S. N., Ibrahim, F. B., Zakaria, E. B., Sulaiman, W. S. W., & Kamaluddin, M. R. (2024) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v14-i5/21720 · Cited by 3
“Authors have utilised the Mixed-Method Appraisal Tool (MMAT) by Hong, et al. (2018) for the quality appraisal stage.”
InfluentialA systematic review on grit among nursing college students in South Korea
Shin, H., Hong, M., & Choi, S. (2024) · The Journal of Korean Academic Society of Nursing Education · doi:10.5977/jkasne.2024.30.2.124 · Cited by 2
Abiddin, N. Z., & Talha, M. D. M. (2024) · Revista de Gestão Social e Ambiental · doi:10.24857/rgsa.v18n6-130 · Cited by 1
“Subsequently, the 24 surviving papers undergo the quality evaluation recommended by Hong et al. (2018).”
Backgroundda Luz, E. M. F., Munhoz, O. L., Greco, P. B. T., et al. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7048.4177 · Cited by 1
Kong’ani, L. N. S. (2024) · Conflict Resolution Quarterly · doi:10.1002/crq.21442 · Cited by 1
Out-of-hospital paramedic interactions with people living with dementia: a scoping review
Han, M. X., Ross, L., Hemingway, L., Anderson, D. A., & Gosling, C. M. (2024) · Age and Ageing · doi:10.1093/ageing/afae143 · Cited by 2
Seikaly, K. (2024) · Journal of Web Librarianship · doi:10.1080/19322909.2024.2382689 · Cited by 5
“Questions from the Mixed Methods Appraisal Tool (MMAT) were then used to determine and select the highest quality research (Hong et al., 2018).”
Sihonqo, B., Kuupiel, D., Nxumalo, C. T., Mokgadi, M. U., & Mpofana, N. (2024) · F1000Research · doi:10.12688/f1000research.153848.1 · Cited by 1
“Spirit child” and concomitant paedicides in Africa: a systematic review of the empirical literature
Owusu, E. S. (2024) · International Journal of Comparative and Applied Criminal Justice · doi:10.1080/01924036.2024.2388238 · Cited by 3
“For this study, the Mixed Methods Appraisal Tool (MMAT) was employed to assess the methodological quality of included studies (Hong, et al., 2018a, 2018b, 2018c).”
InfluentialWagner, A., & Reifegerste, D. (2024) · Crisis · doi:10.1027/0227-5910/a000965 · Cited by 1
Magakwe, T. S. S., Hansraj, R., Nkambule, S. J., et al. (2024) · F1000Research · doi:10.12688/f1000research.151839.1 · Cited by 1
Vestibular-Evoked Myogenic Potential in Otosclerosis: A Systematic Review
Pinto, A. P., Gunjawate, D. R., Malik, R., et al. (2024) · Ear Nose & Throat Journal · doi:10.1177/01455613241278754 · Cited by 1
Videoconferencing support groups for people affected by dementia: a systematic narrative review
Linder, B., Atherton, H., MacArtney, J. I., & Dale, J. R. (2024) · Aging & Mental Health · doi:10.1080/13607863.2024.2414049 · Cited by 3
“The quality of included studies was assessed using the Mixed-Methods Appraisal Tool (MMAT) checklist which is compatible with qualitative, quantitative, and mixed methods research designs (Hong et al., 2018).”
Supportive interventions for family caregivers of adults with mental illness: An integrative review
Saleem, M. M., Khraim, F., Nelson, D., & Forgrave, D. (2024) · Qatar journal of public health. · doi:10.5339/qjph.2024.14 · Cited by 4
Blaquera, A. P., Yukari, H., Takase, K., Ito, H., Yasuhara, Y., Soriano, G. P., Baua, M. E., Papadopoulos, I., & Tanioka, T. (2024) · The Journal of Medical Investigation · doi:10.2152/jmi.71.197 · Cited by 2
Experiences in the tourism industry after spinal cord injury: A scoping review
Greenwood, J., Gustafsson, L. A., & Walder, K. (2024) · Journal of Spinal Cord Medicine · doi:10.1080/10790268.2024.2401650 · Cited by 1
“Tasiemski, Kennedy (26) did not state how they accounted for confounding factors, and Hua and Cole (25) had unclear details about participant recruitment and the role of researchers, therefore selection and researcher bias are unknown.”
Abdi, F., Alimoradi, Z., Ardabili, N. S., Shirdel, E., Rahimi, F., Mirzadeh, N., & Rahnemaei, F. A. (2024) · Middle East Journal of Rehabilitation and Health Studies · doi:10.5812/mejrh-151847 · Cited by 1
Boakye, D. S., Xatse, E. A., Akorli, V. V., Adjorlolo, P. K., Battanis, Y. Y., Frempong, A. B., Ocansey, L. K., Yeboah, C., & Adjorlolo, S. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-088433 · Cited by 2
Jumiyanti, K. R., Ashar, K., Ekawati, M., & Badriyah, N. (2024) · Journal of Ecohumanism · doi:10.62754/joe.v3i8.5570 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT) version 2018 assesses the methodological quality of retained articles (Hong et al., 2018).”
Sanchez, J. M. P., Picardal, M. T., & Fernandez, S. R. (2025) · Internet Reference Services Quarterly · doi:10.1080/10875301.2025.2450339 · Cited by 2
“…using well-known instruments, such as the Critical Appraisal Skills Programme appraisal checklists developed in 1993 (CASP, 2024), the Joanna Briggs Institute critical appraisal tools published in 2005 (JBI, 2024), and the Mixed Methods Appraisal Tool (MMAT) revised in 2018 (Hong et al., 2018).”
Alimento como possível transmissor de SARS-CoV-2: uma revisão integrativa
Pereira, S. C. L., Costa, B. V. D. L., Bento, I. C., et al. (2025) · Nutrivisa Revista de Nutrição e Vigilância em Saúde · doi:10.52521/nutrivisa.v12i1.14428 · Cited by 1
Group learning contracts in healthcare education: A systematic review
Chau, M. T., Singh, C. L., & Bowman, T. (2025) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2024.101843 · Cited by 1
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.