The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,901–3,000 of 4,578
Rault, A., Dolbeault, S., Terrasson, J., Bouleuc, C., Cottu, P. H., Piperno-Neumann, S., Rodrigues, M. J., Vaflard, P., & Brédart, A. (2024) · Pilot and Feasibility Studies · doi:10.1186/s40814-024-01543-y · Cited by 1
Discourse Analysis in End-of-Life Communication: A Systematic Review
Jia, N., & Wu, Y. (2024) · Health Communication · doi:10.1080/10410236.2024.2400815 · Cited by 3
“The quality of the included articles was appraised by two independent reviewers using the Mixed Method Appraisal Tool (MMAT) (Hong et al., 2018) due to the diverse range of study designs (i.e., qualitative, quantitative, and mixed-method studies).”
Associations Between Hospital Nursing Resources and Breastfeeding Outcomes: A Narrative Review
Mason, A., Lake, E. T., & Clark, R. R. S. (2024) · The Journal of Perinatal & Neonatal Nursing · doi:10.1097/jpn.0000000000000859 · Cited by 2
Availability of residents and preceptors for interprofessional practices: mixed methods study
Maffissoni, A. L., Martini, J. G., Lazzari, D. D., Vendruscolo, C., Sanes, M. D. S., & Bresolin, P. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7374.4395 · Cited by 2
Husaini, M. A., Searle, A., & Chuter, V. (2024) · Journal of Foot and Ankle Research · doi:10.1002/jfa2.70009 · Cited by 2
Harm reduction for perinatal cannabis use: protocol for a scoping review of clinical practices
Albanese, A. M., Littlewood, A., Creamer, A., Rogers, B. G., & Elwy, A. R. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-090453 · Cited by 1
“41 42 The protocol, and any potential revisions, has been registered and can be accessed through the Open Science Framework (OSF Registries) (https://osf.io/wb3jc).”
Jiménez-Ros, A. M., Marques, B., Pardo, G. D., & Martins, A. T. (2025) · Actas Españolas de Psiquiatría · doi:10.62641/aep.v53i1.1822 · Cited by 1
“In this review, we used the latest version of MMAT, consisting of 25 items rated on a nominal scale (Yes/No/Do not know) [21].”
Vacchiano, M., Fernandez, G., & Fernandez-Garcia, J. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-089828 · Cited by 1
Long, S., Madon, Z., Norowi, N. M., & Ang, M. F. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1431996 · Cited by 4
Samel, K., Binaut, L., & Bouteyre, É. (2025) · Child Protection and Practice · doi:10.1016/j.chipro.2025.100097 · Cited by 1
Githaiga, J. N., Murphy, C. F., Graham, J., Day, S., Khurana, V., Khosaf, R., Bates, O. B., Besigye, I. K., Weakliam, D., Mash, P. B., et al. (2025) · HRB Open Research · doi:10.12688/hrbopenres.14039.1 · Cited by 2
Hodge, G., Burrows, L., Page, G., & Pearce, S. (2025) · Journal of Long-Term Care · doi:10.31389/jltc.277 · Cited by 1
O’Meara, A., Lewis, C., & Davies, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318428 · Cited by 2
Cooper, J., Wort, G. K., Gillison, F., & Lambert, J. D. (2025) · Mental health and physical activity · doi:10.1016/j.mhpa.2025.100674 · Cited by 4
Ageing of adults who are blind: A scoping review
Honingh, A., Kok, A., Mesker, M. (., Ket, J. C. F., Olsman, E., Veneberg, B., & Sterkenburg, P. S. (2025) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13472 · Cited by 1
“33 This was done independently by two researchers (AKH and MM).”
Black, C. G., Williams, J., Sobel, N., et al. (2025) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-025-02337-5 · Cited by 1
Music therapy and silence – Silence in music therapy. A systematic review
Stolterfoth, C., & Pfeifer, E. (2025) · The Arts in Psychotherapy · doi:10.1016/j.aip.2025.102286 · Cited by 2
Le, H. N., Sofija, E., Harris, N., Noviasty, R., Nguyen, T., & Phung, H. N. (2025) · Food and Energy Security · doi:10.1002/fes3.70065 · Cited by 2
A systematic integrative review of international policies for women experiencing homelessness
Sanantonio, A. G., Botija, M., & Carbonell, Á. (2025) · International Journal of Social Welfare · doi:10.1111/ijsw.70005 · Cited by 3
Nunn, J. L., Takashima, M. D. L. R., Wray-Jones, E. M., et al. (2025) · European Journal of Cancer Care · doi:10.1155/ecc/9956694 · Cited by 1
“Te MMAT tool [24] was used to assess the quality of the studies, and overall, the quality of the studies included was mixed, as summarized in Figure 2.”
Factors Influencing the Quality of Virtual Reality in Nursing Education: A Systematic Review
Hu, Q., Chen, H., Wang, C., Li, S., Cao, H., & Jiang, Y. R. (2025) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001287 · Cited by 1
Bacsa-Károlyi, B., & Fehérvári, A. (2025) · Iskolakultúra · doi:10.14232/iskkult.2025.2.20 · Cited by 2
Santero, M., & Menai, S. D. (2024) · Colombia medica · doi:10.25100/cm.v55i4.6597 · Cited by 5
Lee, L., Nicholson, P., Gerber, K., Naicker, R., & Hutchinson, A. M. (2025) · Campbell Systematic Reviews · doi:10.1002/cl2.70023 · Cited by 1
“This will be assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al. 2018) for primary sources and the Critical Appraisal Skills Programme (CASP) Systematic Reviews Checklist (2018) for secondary sources.”
Jing, W., Gin, L. L., & Subhi, N. (2025) · Sexuality & Culture · doi:10.1007/s12119-025-10351-2 · Cited by 2
Gasteiger, N., Davies, A., Khan, N., Vercell, A., Dowding, D., Ali, S. M., & Davies, A. C. (2025) · Journal of Community Genetics · doi:10.1007/s12687-025-00789-0 · Cited by 1
“Literature was retrieved from eight databases and quality was assessed using the Mixed Methods Appraisal Tool (Hong et al. 2018).”
Putri, O. Q., Akinreni, T., & Kurniawan, A. L. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-088915 · Cited by 1
Harley, I. E. (2025) · Mental Health Practice · doi:10.7748/mhp.2025.e1734 · Cited by 1
Shieu, B. M., Cho, Y.-M., Wang, M., Reid, M. C., Epps, F. R., Li, Y., & Song, L. (2025) · Advances in Integrative Medicine · doi:10.1016/j.aimed.2025.100494 · Cited by 1
Building inclusive rehabilitation education: A scoping review protocol on EDI approaches and impact
Kokorelias, K. M., Chau, V., Boutros, H. M., Wijekoon, S., Singh, H., Kalu, M. E., & Harris, M. T. (2025) · PLoS ONE · doi:10.1371/journal.pone.0322110 · Cited by 1
“To enhance the rigor and generalizability of our findings, we will apply the Mixed Methods Appraisal Tool (MMAT) to assess the quality of included studies, particularly those employing mixed-methods approaches [28].”
Social Needs and Type 2 Diabetes in Latinos: An Integrative Review
Rascón, A. M., Owolabi, E. O., Braxton, M. E., Verdecias, N., & Shaibi, G. Q. (2025) · Journal of Racial and Ethnic Health Disparities · doi:10.1007/s40615-025-02448-z · Cited by 2
“Level of evidence was determined using the Mel-nyk Hierarchy of Evidence Model [18] and quality of evidence was evaluated by two independent reviewers using the Mixed Methods Appraisal Tool (MMAT) version 18 [19].”
Lau, T. W., Tse, M., Liu, Y., & Leung, A. Y. M. (2025) · Australian Journal on Ageing · doi:10.1111/ajag.70034 · Cited by 1
“31 Quality assessment identified a good fit between research designs and questions.”
Stansfield, L., Sénécal, A.-M., He, S., et al. (2025) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2025.2501997 · Cited by 1
Gains, A. M., Moghaddam, N. G., Baliousis, M., & Biswas, S. (2025) · BMJ Supportive & Palliative Care · doi:10.1136/spcare-2024-005342 · Cited by 2
“31 The MMAT was selected due to suitability for mixed methods designs, as was expected for research within VR, and as this tool is considered to have sufficient content validity.”
Giardulli, B., Pagnucci, N., Przyłęcki, P., Koutra, K., Walsh, N., Androulakis, C., Aleo, G., Tziraki, C., Testa, M., & Battista, S. (2025) · Journal of Interprofessional Care · doi:10.1080/13561820.2025.2495018 · Cited by 3
“The methodological quality of the included studies was assessed by one researcher (PP) adopting the Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018).”
The role of digital health interventions in chronic ulcer management: a systematic review
Stevenson, E. M., Coda, A., Chu, G., & Bourke, M. (2025) · Journal of Tissue Viability · doi:10.1016/j.jtv.2025.100928 · Cited by 2
Abaraogu, U. O., Dall, P., Seenan, C., Rhodes, S., Gorely, T., McParland, J. L., Brittenden, J., Anieto, E. M., Booth, L., Gormal, C., et al. (2025) · Health Technology Assessment · doi:10.3310/zbng5240 · Cited by 1
Shah, P., Stolt, M., & Metsälä, E. (2025) · Acta Radiologica · doi:10.1177/02841851251344475 · Cited by 1
How to assess the quality of research in nursing
Mannethodi, K., & Nashwan, A. J. J. (2025) · Evidence-Based Nursing · doi:10.1136/ebnurs-2025-104355 · Cited by 1
“11 Use case: ideal for systematic reviews that include diverse study designs.”
Más que una foto fija: revisión sistemática integrativa sobre la medición del sinhogarismo en España
Gallén-Granell, E., Botija, M., & Sanantonio, A. G. (2025) · Revista iberoamericana de estudios de desarrollo = Iberoamerican journal of development studies · doi:10.26754/ojs_ried/ijds.11169 · Cited by 1
Dod, J., & Lansdown, G. (2025) · British Paramedic Journal · doi:10.29045/14784726.2025.6.10.1.38 · Cited by 1
“Review papers or secondary data – only primary data were used to ensure quality and minimise bias. for quality using the CASP critical appraisal tool (CASP, 2024) for qualitative studies and the MMAT for mixed method studies (Hong et al., 2018).”
Manias, E., Jarden, R., Ottosen, K., et al. (2025) · Australian Critical Care · doi:10.1016/j.aucc.2025.101267 · Cited by 1
Pitt, C., Ullman, A. J., & Bradford, N. K. (2025) · Journal of Child Health Care · doi:10.1177/13674935251351070 · Cited by 1
“Included in the MMAT are algorithms that can assist in identifying the study design and therefore allowing the reviewer to apply the appropriate critique (Hong et al., 2018).”
Compassion-focused approaches in forensic and inpatient mental health services: a systematic review
Waite, H. O. H., Wane, J., & Lamers, C. (2025) · Journal of Forensic Practice · doi:10.1108/jfp-11-2024-0065 · Cited by 1
Liew, Y., Brown, L., Madden, R., Alston, L. V., Urquhart, L., Heaney, S. E., & Schumacher, T. L. (2025) · Current Nutrition Reports · doi:10.1007/s13668-025-00674-9 · Cited by 2
Nweke, M., Govender, N., Nweke, I., & Pillay, J. D. (2025) · MethodsX · doi:10.1016/j.mex.2025.103494 · Cited by 1
“The potential bias in the eligible individual studies will be evaluated using the mixed method appraisal tool (MMAT) [33], which includes five domains.”
Borji‐Navan, S., Salehin, S., NaseriBooriAbadi, T., Goli, S., & Mirghafourvand, M. (2025) · Health Science Reports · doi:10.1002/hsr2.70977 · Cited by 1
Wallace, T., Koebbel, C., & Heath, J. (2025) · Journal of Health Psychology · doi:10.1177/13591053251351222 · Cited by 5
“The mixed meth-ods study was appraised using the Mixed-Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Use of Simulation in Nonclinical Graduate Nursing Programs: An Integrative Review
Patel, D. A., Watts, P. I., & Billings, R. (2025) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001357 · Cited by 1
A novel approach – occupational therapy on campus: A systematic review
Keptner, K. M., Avendano, K., Eusebio, B. J., Kern, Z., Pekala, M., & Vincent, E. (2025) · Journal of American College Health · doi:10.1080/07448481.2025.2533913 · Cited by 3
Self-esteem in gifted adolescents: A systematic review and bibliometric analysis
Yüzbaşıoğlu, F., & Talu, E. (2025) · High Ability Studies · doi:10.1080/13598139.2025.2529898 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT), developed by Hong et al. (2018), was used to evaluate the risk of bias and quality.”
Raihan, A. S., Islam, F., Nimbarte, A. D., & Ahmed, I. (2025) · Journal of Environmental Assessment Policy and Management · doi:10.1142/s1464333225300020 · Cited by 2
Conversational Agents for Managing Chronic Diseases: A Systematic Review
Zakerabasali, S., Amiri, P., & Alipour, J. (2025) · Health Scope · doi:10.5812/healthscope-163131 · Cited by 1
Indigenous communities and mining activities in Central America and Mexico: A systematic review
Leroy, D. (2025) · The Extractive Industries and Society · doi:10.1016/j.exis.2025.101747 · Cited by 4
Aslan, P., & Cheung, M. (2025) · Sociology Compass · doi:10.1111/soc4.70106 · Cited by 2
Wen, B., Bola, R., Scott, T. M., et al. (2025) · Journal of surgical education · doi:10.1016/j.jsurg.2025.103659 · Cited by 1
Tang, Q., & Kamarudin, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-2170-7.ch014 · Cited by 5
Balla, S. B., Angelakopoulos, N., Tadakamadla, J., & Tadakamadla, S. K. (2025) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.70021 · Cited by 4
“The quality of evidence was assessed using the MMAT tool, indicating that most studies are of moderate to high methodological quality [31].”
Hill, C. L., Namara, K. M., Alston, L. V., Miles, A., Choi, J., & Shee, A. W. (2025) · International Journal of Speech-Language Pathology · doi:10.1080/17549507.2025.2546001 · Cited by 1
Developing a scale to measure social dynamics in funeral ceremonies: A mixed methods approach
Esmailnia, S., Albuquerque, S., & Coelho, A. (2025) · Death Studies · doi:10.1080/07481187.2025.2559726 · Cited by 2
Acute and prophylactic treatment of dialysis headache: a systematic review
de Sousa, J. R., Santana, M. C. A., & Rocha‐Filho, P. A. S. (2025) · Expert Review of Neurotherapeutics · doi:10.1080/14737175.2025.2562123 · Cited by 1
“…(RoB 2) for randomized clinical trials [13], ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions) for non-randomized studies with comparator interventions (cohort and case-control studies) [14], and MMAT (Mixed Methods Appraisal Tool, version 2018) [15] for the other included studies.”
Cayetano, A. R., Muñoz, S. P., Simón, D. N., & Morales-Campo, P. T. (2025) · Education Sciences · doi:10.3390/educsci15101282 · Cited by 4
Kairuz, C. A., Hunter, K., Barnier, B., Porykali, B., Ford, B., Bennett‐Brook, K., Mackean, T., Litton, E., Affandi, J. S., Ryder, C., et al. (2025) · BMC Medical Ethics · doi:10.1186/s12910-025-01280-8 · Cited by 2
“This is a quality appraisal tool for reviews that include qualitative, quantitative and mixed methods studies [32].”
Hughes, M. H., Paturzo, J. G. R., Fernandez, C., Garg, K. S., Rajan, J., Rahematpura, S., DiBello, J. R., Shaikh, U., Wrigley, J., Phillips, N. S., et al. (2025) · Pediatric Blood & Cancer · doi:10.1002/pbc.32072 · Cited by 1
“The 2018 Mixed Methods Appraisal Tool (MMAT) was used to conduct a quality assessment of all included studies [32].”
Kok, M. O., Fanda, R. B., Lubbers, R., van Gurp, M., Ravinetto, R., & Probandari, A. N. (2025) · The Journal of Medicine Access · doi:10.1177/27550834251371502 · Cited by 3
“45,46 Developed to evaluate qualitative, quantitative and mixed methods research, the MMAT is specifically designed for reviews that include diverse study designs.”
Tong, K., Wainwright, J., Horne, J. K., Jones, K., Daďová, K., Guggiari, E., Leu, A., Birkett-Swan, L., & Všetečková, J. (2025) · Aging & Mental Health · doi:10.1080/13607863.2025.2564719 · Cited by 3
“The quality of the identified articles was assessed using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018), which has a specific matrix applicable to qualitative, quantitative mixed-methods studies, and mixed-methods studies, respectively.”
Chalk, A., Bauer, A., Higgins, N., Tranter, I., & Nitz, M. (2025) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/00048674251374475 · Cited by 1
“Quality assessment was performed using the Mixed Method Quality Assessment Tool (Hong et al., 2018; MMAT Version 2018).”
Ola, O. S., & Lizam, M. (2025) · Multidisciplinary Reviews · doi:10.31893/multirev.2026182 · Cited by 2
Immersive technology applications in radiation protection for fluoroscopy - A systematic review
Videira, S., Rodrigues, M. A., Santos, J., & da Silva (2025) · Radiography · doi:10.1016/j.radi.2025.103179 · Cited by 2
Nygård, S., Nysæter, M. K., Ferreira, L. R., Frøyland, L. R., & Kvalem, I. L. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251375912 · Cited by 3
Haenen, A. I., Frielink, N., van Ool, J. S., Snoeijen‐Schouwenaars, F. M., & Embregts, P. J. C. M. (2025) · Journal of Intellectual Disability Research · doi:10.1111/jir.70051 · Cited by 4
“To provide an overview of the quality of existing research, we assessed the methodological quality of the included studies by using the Mixed Methods Appraisal Tool (MMAT; Hong et al. 2018).”
Arjun, B., Chaitra, C. M., Ananthakrishnan, M., et al. (2025) · Cureus · doi:10.7759/cureus.94143 · Cited by 1
“The quality appraisal of the included studies was conducted using the Mixed Methods Appraisal Tool (MMAT) [12].”
Mohamad, D., Pesonen, H. V., Keleş, S., et al. (2025) · International Journal of Inclusive Education · doi:10.1080/13603116.2025.2567417 · Cited by 1
Hayes, C. C., Najm, D., Vasilevski, V., Sweet, L. P., & Alexander, L. (2025) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2025.101813 · Cited by 5
Brain connectivity changes associated with acute sleep deprivation: a systematic review
Hosang, S., Wong, K., Mlinarevic, D., Krivoruk, B., Ravi, H., Hamilton, D., Norton, L. E., Stojanoski, B. B., & Ganesan, S. L. (2025) · Neurological Sciences · doi:10.1007/s10072-025-08517-9 · Cited by 4
“The Mixed Methods Appraisal Tool (MMAT) was used to assess the quality of studies and their risk of bias [13].”
Wahid, S., Becht, A., Eltanamly, H., Thomaes, S. C. E., & Branje, S. (2025) · Identity · doi:10.1080/15283488.2025.2564091 · Cited by 3
Dami, A., Mahé-Duverger, É., Acien, M., Blanchette, V., & Moisan, G. (2025) · Prosthetics and Orthotics International · doi:10.1097/pxr.0000000000000486 · Cited by 1
Lai, M., Jian, M., & Wang, M. (2025) · The British Journal of Social Work · doi:10.1093/bjsw/bcaf228 · Cited by 7
Delaney, A. L., Flatt, A., Koepp, H., Fial, A., & Hustad, K. C. (2025) · American Journal of Speech-Language Pathology · doi:10.1044/2025_ajslp-25-00066 · Cited by 2
“Fourteen studies met inclusion criteria and were rated using the Mixed Methods Appraisal Tool (Hong et al., 2018) for quality assessment.”
Rural digital social innovation for health and social care: A systematic review
Li, E. P. H., Kushnerik, T., Seaton, C. L., Rush, K. L., Aulakh, P., Zajko, M., Hasan, K., Manhas, R., Yakong, V. N., & Janke, R. (2025) · SSM - Health Systems · doi:10.1016/j.ssmhs.2025.100143 · Cited by 3
Bruun, A., Thurgate, C., & Joshua, B. (2025) · Journal of Further and Higher Education · doi:10.1080/0309877x.2025.2573016 · Cited by 1
“Critical appraisal of included studies was conducted using the Mixed Methods Appraisal Tool (MMAT), version 2018 (Hong et al. 2018).”
Lokeesan, V., Finlayson, K. J., Edwards, H. E., & Parker, C. N. (2025) · International Journal of Nursing Practice · doi:10.1111/ijn.70080 · Cited by 2
Cano-Cañada, E., Tapia‐Serrano, M. Á., Saiz-González, P., Fernández‐Río, J., & Gallego, D. I. (2025) · European Physical Education Review · doi:10.1177/1356336x251385414 · Cited by 6
“Two researchers (ECC and MATS) assessed the quality of the review using the Mixed Methods Assessment Tool (MMAT; Hong et al., 2018), with feedback provided by a third researcher (DI).”
Khosravi, M. (2025) · International Journal of Bipolar Disorders · doi:10.1186/s40345-025-00400-y · Cited by 1
“Quantitative studies were assessed using the Newcas-tle-Ottawa Scale (Carra et al. 2025), and mixed-methods studies were appraised using the Mixed Methods Appraisal Tool (Hong et al. 2018).”
Ofori-Parku, C. Y., Harris, O., Baltzell, K., Little, E., & Asiodu, I. V. (2025) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-025-08247-y · Cited by 1
“Quality assessment was completed at the study level using the MMAT (Hong et al., 2018).”
InfluentialMaqsood, A. (2025) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205251389360 · Cited by 1
“26,27 Studies were not omitted only based on quality assessment; rather, the robustness of each study ’ s contribution was evaluated during the synthesis of data and the rating of con fi dence in results.”
Samarasinghe, B. S. W., Millar, R. W., Exworthy, M., & Aunger, J. A. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13592-x · Cited by 1
Kerkove, K. S., & Culross, B. (2025) · Journal of Nursing Education · doi:10.3928/01484834-20250626-01 · Cited by 4
Bers, M. U., Alrawashdeh, G. S., Nievera, M. A., & Nadler, E. (2025) · Journal of Moral Education · doi:10.1080/03057240.2025.2569883 · Cited by 2
The Mental Health Profile of Borderline Intellectual Functioning: A Systematic Review
Cho, M., Ko, E., Choi, H., Han, D.-G., Kim, H. J., Kim, B., & Park, M.-H. (2025) · Psychiatry Investigation · doi:10.30773/pi.2025.0209 · Cited by 1
“The quality of the included studies was assessed according to the Mixed Methods Appraisal Tool (MMAT), V.2018, 18 which was designed for assessing reviews including articles of various study designs.”
Rhodes, M., Milne, C., Pellizzer, M. L., & Prichard, I. J. (2025) · Body Image · doi:10.1016/j.bodyim.2025.101998 · Cited by 3
Jiang, Z., & Chiang, F. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100745 · Cited by 3
Serenity Room as a Healing Space for Nurses: A Mixed-Method Systematic Review
Primavera, E., Fiorini, J., Bella, V. D., Trebbi, E., Tognaccini, L., Zaghini, F., & Sili, A. (2025) · Journal of Holistic Nursing · doi:10.1177/08980101251395787 · Cited by 2
Scholes, A. K., Elma, A., Cernat, A., Dawe, R. E., Grierson, L., Hamza, D. M., O’Toole, D., Scott, I. M., Shen, K., Shuldiner, J., et al. (2025) · Family Practice · doi:10.1093/fampra/cmaf092 · Cited by 4
“Consistent with our review methodology, the MMAT tool, and our previously published stance on critical appraisal, all eligible studies were included, as long as they presented data to support their conclusions [13, 17, 18].”
Stakeholder perceived value of telehealth: a systematic review
Sumanasekera, K., & Todorova, N. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-105702 · Cited by 1
Zeng, X., Card, A., Choi, N. G., & Zhou, Y. (2025) · Clinical Interventions in Aging · doi:10.2147/cia.s553910 · Cited by 2
“Two authors (X.Z. and A.C.) independently reviewed all full-text articles and extracted data into an Excel spreadsheet, following the PRISMA 2020 systematic review protocol.”
Ogunjobi, D. T., Abu, S., Ajayi, A. J., Emenike, M. I., Obong, E. E., & Mensah, T. K. (2025) · Australian Journal of Biomedical Research · doi:10.63946/aubiomed/17464 · Cited by 2
Community engagement in mass drug administration participatory interventions: A scoping review
Chouaïd, A., Louart, S., Faye, A., Bâ, E. H., Landier, J., & Ridde, V. (2025) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0013737 · Cited by 1
Mercer, F., McIntyre, K., Lawson, M., & O’Rourke, S. J. (2025) · Child & Family Social Work · doi:10.1111/cfs.70096 · Cited by 1
Self-Care of Family Caregivers of People Peritoneal Dialysis
Escamilla, M. D. P. V., Saavedra, M. O., & Aguilar, A. E. (2025) · Salud Ciencia y Tecnología - Serie de Conferencias · doi:10.56294/sctconf20251764 · Cited by 1
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.