The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 3,401–3,500 of 4,578
Lim, C. Y., Chan, P. Y., & Shorey, S. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16392 · Cited by 1
A scoping review on control strategies for Echinococcus granulosus sensu latoPreprint
Cruz-Saldaña, T. D. L., Bustos, J. A., Requena-Herrera, M. P., Balarezo, N. M. M., Ortiz-Cam, L., Caceres, A.-L., Guzman, C., Gavidia, C. M., Ugarte‐Gil, C., & Castillo-Neyra, R. (2024) · medRxiv · doi:10.1101/2024.08.21.24312335
Farias, A. N., Pucinelli, R. H., Ramos, R. C., Fernández‐Río, J., & Impolcetto, F. M. (2024) · Caderno Pedagógico · doi:10.54033/cadpedv21n8-267 · Cited by 2
Munhoz, O. L., Morais, B. X., da Luz, E. M. F., Greco, P. B. T., Ilha, S., & Magnago, T. S. B. D. S. (2024) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-0347en · Cited by 4
Munhoz, O. L., Morais, B. X., da Luz, E. M. F., et al. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7218.4276
Munhoz, O. L., Morais, B. X., da Luz, E. M. F., et al. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7218.4274
Lim, H.-A., & Song, Y. (2024) · Journal of Health Informatics and Statistics · doi:10.21032/jhis.2024.49.3.205
Krishnan, A., & Deb, A. (2024) · Current Psychology · doi:10.1007/s12144-024-06650-w
“The qualitative, quantitative, and mixed method articles were screened using the Mixed Methods Appraisal Tool, Version 2018, by Hong et al. (2018).”
Abasi, A., Ayatollahi, H., & Amirzadeh, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4870800/v1
RetractedThe Role of Spiritual Leadership in Cultivating Organizational Synergy: A Systematic Review
Jahan, A., Ali, Z. F. B., & Usman, R. (2024) · Bulletin of Business and Economics (BBE) · doi:10.61506/01.00468
“Quality Review Included articles were each reviewed independently twice for methodological quality by the researcher using an adapted quality assessment tool “The mixed-method appraisal tool version 2018” designed by (Hong, 2018).”
MethodAssistance of emergency and urgent care services to people attempting suicide: integrative review
de Oliveira, N. G., Ruela, S. F., de Souza, T. L., et al. (2024) · Nursing & Care Open Access Journal · doi:10.15406/ncoaj.2024.10.00298
Masemola, H. C., Nkambule, S. J., Baloyi, O. B., & Xulu-Kasaba, Z. N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305904 · Cited by 3
Ayandeyi, K., Shaha, S., & Chilaka, M. A. (2024) · Journal of Family Medicine and Health Care · doi:10.11648/j.jfmhc.20241003.14
Shared Decision-Making in Neurologic Palliative Care and End of Life
Ben‐Zacharia, A. B., Bartels, A., & Brugger, H. T. (2024) · Physician Assistant Clinics · doi:10.1016/j.cpha.2024.08.003
Shukla, A., Godhamgaonkar, A., Gadikoppula, R., et al. (2024) · doi:10.2196/preprints.67054
Collaborative design of IoT systems for smart cities with and for citizen participation
de Alencar, T. S., & Zaina, L. A. M. (2024) · Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais (IHC) · doi:10.5753/ihc_estendido.2024.241666
Alhakbani, M. S. A., AlQahtani, A., AlTreef, W. A., Aleisa, A. I., Gahtani, H. K. A., & Al-Nasser, M. N. (2024) · Cureus · doi:10.7759/cureus.71732
“Studies were classified as either “low (score ≤3)” or “high (score >3)” depending on the reviewer’s response: “yes” (1 point) or “no” (0 points) [28].”
Maffissoni, A. L., Martini, J. G., Lazzari, D. D., Vendruscolo, C., Sanes, M. D. S., & Bresolin, P. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7374.4396 · Cited by 2
PREVALENCE AND RISK FACTORS OF LUMBOPELVIC PAIN AMONG IRANIAN PREGNANT WOMEN: A SYSTEMATIC REVIEW
Shamsipour, S., & Akbari-Chehrehbargh, Z. (2024) · Nursing and Midwifery Journal · doi:10.61186/unmf.22.6.451
Parlak, L., Duru, P., & Örsal, Ö. (2024) · Ordu Üniversitesi Hemşirelik Çalışmaları Dergisi · doi:10.38108/ouhcd.1379709 · Cited by 1
Appraised evidence and decision making in clinical practicePreprint
Leliopoulou, C., Stroumpouki, T., Stafylarakis, E., & Collins, L. (2024) · medRxiv · doi:10.1101/2024.11.15.24317373 · Cited by 1
Dahmash, D. T., Michelen, M., Rigby, I., Piotrowski, H., Nartowski, R. O., Cheng, V., Dagens, A., Harriss, E., Hart, P. J., Jacob, S. T., et al. (2024) · medRxiv · doi:10.1101/2024.11.21.24317702 · Cited by 1
Hasan, M. Z., Webb, E., Quayyum, Z., & Ensor, T. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02714-x · Cited by 1
“The MMAT is a general tool that evaluates quantitative, qualitative, and mixed-methods studies [25].”
Qualidade de Vida de Pacientes com Câncer Avançado em Tratamento: Revisão e Síntese Qualitativa
Hansen, C. D. A. P., Nunes, C. R. D. O., & Fernandes, T. A. S. (2024) · Revista Brasileira de Cancerologia · doi:10.32635/2176-9745.rbc.2024v70n4.4835
Akbari‐Shahrestanaki, Y., Aminizadeh, M., Shahrebabak, A. G., et al. (2024) · Journal of Education and Health Promotion · doi:10.4103/jehp.jehp_1080_23
Jamaludin, T. S. S., Nurumal, M. S., Sukeri, S. H. M., Hasan, M. K. C., Win, K. K., & Noviani, W. (2024) · INTERNATIONAL JOURNAL OF CARE SCHOLARS · doi:10.31436/ijcs.v7i3.390 · Cited by 2
A Comprehensive Review of Traditional Irrigation Systems: Sustainability and Future ProspectsPreprint
Branco, C. C. R.-D.-S., de Brito, A. G., & Seixas, P. C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5043719
Mentoring Matters: Evaluating The Black Physicians of Canada Mentorship ProgramPreprint
Obetta, C., Menezes, A., Chandran, N., Egbedeyi, O., Taghavi, M., Tunde‐Byass, M., Yu, C. H. Y., Kalocsai, C., Najeeb, U., Swaleh, R. M., et al. (2024) · medRxiv · doi:10.1101/2024.12.07.24318652 · Cited by 3
Saga, M., Rkhaila, A., Ghros, S. G., & Ounine, K. (2024) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/979-8-3693-3707-3.ch003
Wenje, S., Calvert, S., Henshaw, H., Spriggs, R. V., Dening, T., Hendron, E., & Heffernan, E. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-087418
“61 It encompasses five methodological domains: qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies and mixed methods studies.”
Ebersöhn, L., Murphy, P. K., & Basson, L.-M. (2024) · Jornal de Políticas Educacionais · doi:10.5380/jpe.v18i1.96061
Hong, Y. J., & Chang, S. O. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16706
Amofah, A. D., Apaflo, J. N., & Opoku, L. K. (2024) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2024.24.3.3950 · Cited by 1
“The Mixed Methods Appraisal Tool (MMAT) [11] was used for quality assessment.”
First responders are exposed to high levels of trauma, are affected by it, and need support
Molodynski, A. (2025) · Evidence-Based Nursing · doi:10.1136/ebnurs-2024-104202
Gasteiger, N., Norman, G., Grainger, R., van der Veer, S. N., McGarrigle, L., Jones, D. J., Eost‐Telling, C., Vercell, A., Ford, C., Ali, S. M., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae159 · Cited by 1
“35 Other tools such as ROBIS, 36 the CASP Systematic Review Checklist, 37 or AMSTAR 2 13 are used to critically appraise systematic reviews.”
Ottwein, J. K. (2022) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2048603
The use of coaching in stroke rehabilitation: a scoping review
Kessler, D., & Franz, M. (2025) · Disability and Rehabilitation · doi:10.1080/09638288.2025.2455528
“Methodological quality of the studies was evaluated by the primary author using the Mixed Methods Appraisal Tool (MMAT) [22] to give an overall impression of the methodological strength of the literature. each study was rated on five criteria relevant to the study design.”
Ali, A., Pirie, K., Hughes, C., Jacob, B. M., Doyle, F., Brain, K. E., Bonniec, A. L., Cotter, S., Walsh, K. R., McCutchan, G. M., et al. (2025) · HRB Open Research · doi:10.12688/hrbopenres.13931.1
Rehill, N., Halvorsrud, K., Shand, J., Fonagy, P., & Raine, R. (2025) · Public health reviews · doi:10.3389/phrs.2024.1607955
“Quality assessment of each included study was conducted independently by NR and KH using the Mixed Methods Assessment Tool [21], chosen for its fl exibility across a range of study designs, and disagreements discussed until consensus was reached.”
Household cleaning agents impact on pediatric asthma: a systematic review and meta-analysis
Arif, M. I., Wang, Z. Y., & Ru, L. (2025) · International Journal of Environmental Health Research · doi:10.1080/09603123.2025.2467820 · Cited by 1
“Studies were classified as either low when scored ≤ 3 or high when scored > 3 (Hong et al. 2018) and whole process of quality assessment was performed by two authors.”
Breese, A., Clark, M., & Halliday, L. E. (2025) · International Journal of Palliative Nursing · doi:10.12968/ijpn.2025.31.2.68
Chu, C. D., Dengel, L. T., Petroni, G. R., Hollen, P. J., & Jones, R. A. (2025) · Oncology nursing forum · doi:10.1188/25.onf.e58-e64
“For the purposes of this integrative review and because of the variability of research methodologies, the current authors used the Mixed Methods Appraisal Tool, version 2018 (Hong et al., 2018).”
Bali, S., Singh, S., & Arora, A. (2025) · Health Problems of Civilization · doi:10.5114/hpc.2025.146537
Shariff, S., Baharudin, S. ‘., & Ismail, N. A. (2025) · Information Management and Business Review · doi:10.22610/imbr.v17i1(i).4350 · Cited by 6
“MMAT allows appraisal of all types of methodologies studies (Hong et al. 2018).”
Psychosocial Health in Adults With Sickle Cell Disease
Adeyinka, S., & Griggs, S. A. (2025) · Journal of Psychosocial Nursing and Mental Health Services · doi:10.3928/02793695-20250217-02
McLean, J., Chambers, P., Steventon, L., Cruickshank, S., & Nabhani‐Gebara, S. (2025) · BMJ Quality & Safety · doi:10.1136/bmjqs-2024-017560
“23 Participant and design characteristics of the studies are summarised in table 2.”
What causes cyberbullying among Asian school students: A systematic literature review
Jamaluddin, A., Isa, N. J. M., Dawi, A. H., et al. (2025) · Salud Ciencia y Tecnología - Serie de Conferencias · doi:10.56294/sctconf20251515
Culture, Acculturation, and Physical Activity among US Hispanic/Latino Adults
Berumen-Flucker, B., Esqueda, K., Ledezma, K., Barrientez, T., Williams, R. D., & Romero, S. (2025) · American Journal of Health Behavior · doi:10.5993/ajhb.49.1.4
A Systematic Review of Virtual Health Assistants in Preventive Cancer Care Communication (Preprint)Preprint
Raisa, A., Chen, X., Bryan, E. G., Bylund, C. L., Alpert, J. M., Lok, B. C. L., Fisher, C. L., Thomas, L., & Krieger, J. L. R. (2025) · doi:10.2196/preprints.73616
A Systematic Review of Interventions for Perinatal Suicidal Behaviour in WomenPreprint
Scarth, B., Derakhshan, E., Witt, K., & Andriessen, K. (2025) · Research Square · doi:10.21203/rs.3.rs-6292915/v1
Carbapenemase genes in hospital wastewater in Africa: A systematic review and meta-analysisPreprint
Hamidou, O., Yacouba, A., Tapha, O., et al. (2025) · medRxiv · doi:10.1101/2025.03.26.25324662
Johnson, R. D. (2024) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2415981
Mus, M., Hadjes, A., Mercier, H., & Chevallier, C. (2025) · Journal for Nature Conservation · doi:10.1016/j.jnc.2025.126910 · Cited by 3
Driving safety among adolescents with health conditions: An integrative review
DiFusco, L. A., Liu, X. (., Curry, A. E., & McDonald, C. C. (2025) · Journal of Transport & Health · doi:10.1016/j.jth.2025.102040 · Cited by 1
Tavares, S., Kuhn, I. L., Harrison, M., et al. (2025) · medRxiv · doi:10.1101/2025.04.08.25325333
Garcia-Garcia, A., Carretero-Randez, C., Camacho‐Bejarano, R., Chicano, M. T. R., Santana, P. R. C., Camacho-Montaño, L. R., Montero‐Marco, J., Charlo‐Bernardos, M., & Orts‐Cortés, M. I. (2025) · Healthcare · doi:10.3390/healthcare13070839
“The Joanna Briggs Institute critical appraisal tool for quasi-experimental studies will be used to assess the quality of quasi-experimental studies, and the Mixed Methods Appraisal Tool (MMAT) scale will be used to assess the quality of mixed-design studies [24].”
Defining Public Health Transformation: A Scoping Review
Phooko, V., Maluleke, K., Ramatsokotla, S., et al. (2025) · Public Health Challenges · doi:10.1002/puh2.70048
Koyanagi, M., Sakuramoto, H., Kajiwara, K., et al. (2025) · Cureus · doi:10.7759/cureus.83029
Zhai, S., Goodman, A., Smith, A. C., Diminic, S., & Zhou, X. (2025) · medRxiv · doi:10.1101/2025.04.27.25326251
Needs of the Parent–Adolescent Dyad With Type 1 Diabetes: A Concept Analysis
Moreno, C., & Carrascal, G. C. (2025) · Nursing Forum · doi:10.1155/nuf/5599983
Kimiecik, C., Reed, J. B., Rodriguez, N. M., Klenosky, D. B., Wadsworth, S. M., Amell, J. W., Moser, M., & Ruiz, Y. (2025) · Journal of Human Behavior in the Social Environment · doi:10.1080/10911359.2025.2501150
Economic evaluation of hospital-to-home transition interventions: a rapid review protocolPreprint
MacEachern, E., Nelson, M. L. A., Leighton, J., Saragosa, M., Graham, I. D., Kokorelias, K. M., Herbert, P., Ludwig-Prout, B., & Thavorn, K. (2025) · medRxiv · doi:10.1101/2025.05.06.25327087
Neonatal Twin Cobedding: A Mixed Method Study of Policy, Practice and Staff Perceptions in Australia
Mann, P., Foster, J. P., Psaila, K., & Schmied, V. A. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.17021 · Cited by 1
“MMAT guidelines (Hong et al. 2018) were used to prepare this manuscript (see Supplementary File 2 ).”
McPhillips, R., Allen, R. E., Ramesh, P., Barnett, K., Chadwick, H., Batool, S., Elahi, A., Hawton, K., Huxley, P. J., Krayer, A., et al. (2025) · Crisis · doi:10.1027/0227-5910/a001004 · Cited by 4
Guo, G., Naqvi, S. Z. F., & Hanchate, N. K. (2026) · BMC Psychiatry · doi:10.1186/s12888-026-08279-4
“The quality of fourteen studies included in this review were appraised using the MMAT [36].”
Global Trends and Outcomes of Research Integration in Medical Curricula: A Systematic ReviewPreprint
Krungkraipetch, L., Krungkraipetch, K., Usimat, K., & Bookaew, S. (2025) · Research Square · doi:10.21203/rs.3.rs-6432347/v1
Tokophobie und Selbstwirksamkeit – welchen Einfluss hat die Geburtsvorbereitung durch Hebammen?
Döring, E. (2025) · Hebamme · doi:10.1055/a-2512-6926
Navigational difficulties after stroke: A systematic review and meta-analysis
Bosmans, J., Moyaert, J., Eetvelde, J. V., & Gillebert, C. R. (2025) · The Clinical Neuropsychologist · doi:10.1080/13854046.2025.2498044
“…& Maurer, 2018), the leiden navigation test (van der Kuil et al., 2022), and the Route learning test (Barrash et al., 2000), whereas for the (hong et al., 2018). given the lack of clear guidelines on correcting for multiple comparisons, all studies that did not apply such corrections were…”
Electronic consent for school-age vaccinations: a rapid reviewPreprint
Okolie, C., Ezenwugo, G., Barry, M., Richmond-Russell, K., & Cottrell, S. L. (2025) · medRxiv · doi:10.1101/2025.05.15.25327678
“The quality of included studies was assessed using three critical appraisal tools – the JBI critical appraisal tool for assessment of risk of bias for randomised controlled trials (Barker et al., 2023), the JBI checklist for quasi-experimental studies (Barker et al., 2024), and the mixed methods appraisal tool (MMAT)…”
InfluentialA Rapid Review of Pathways to Diagnosis for People Living with Peripheral Artery DiseasePreprint
Clothier, Z., Armes, J., Heiß, C., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5255268
Characteristics of critical thinking education to children: a systematic review
Asadi, M., Pouretemad, H., & Zardkhaneh, S. A. (2025) · The Educational and Developmental Psychologist · doi:10.1080/20590776.2025.2499561 · Cited by 5
“The methodological quality of the submitted articles was reviewed by the first and third authors using the Mixed Methods Assessment Tool (MMAT), version 2018 (Hong et al., 2018).”
Li, X., Costa, S., & Giulianotti, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5264567
Hampshire, P. K., & Elizarraras, J. (2025) · Sleep and Vigilance · doi:10.1007/s41782-025-00309-7
“The MMAT was designed for use in systematic reviews that include the appraisal of quantitative, qualitative and mixed methods studies (MMAT [29]).”
Lim, T., Koscak, J., & Swidrovich, J. P. (2025) · Journal of HIV & Social Services · doi:10.1080/15381501.2025.2505845
“Each peer-reviewed article was appraised using the Mixed Methods Appraisal Tool by Hong et al. (2018) as shown in Table 5.”
Gartner, J., Landa, P., Haren, M. T., Lemaire, C., Tànfani, E., Paquet, C., Bergeron, F., & Côté, A. (2026) · Journal of Medical Internet Research · doi:10.2196/78506 · Cited by 1
“Quality assessment was performed using the Mixed Meth-ods Appraisal Tool (MMAT) of July 2020 [33], which uses five core quality criteria to assess each of the following study designs: (1) qualitative, (2) randomized controlled, (3) nonrandomized, (4) quantitative descriptive, and (5) mixed methods.”
Volkmer, A., Alyahya, R. S. W., Atkinson, H. P., et al. (2025) · medRxiv · doi:10.1101/2025.06.11.25329459
Capparozza, M., Eder, T., Wagner, S., & Lachner, A. (2025) · Proceedings. · doi:10.22318/icls2025.340211
Ghiyasvandian, S., Hoseini, A. S. S., Atapour, M., & Moghdam, Z. B. (2025) · Frontiers of Nursing · doi:10.2478/fon-2025-0018
Vasiee, A., Tayyar-Iravanlou, F., Rahmani, F., Zeidani, A., & Gilan, M. S. (2025) · Journal of Aging Research · doi:10.1155/jare/8814069 · Cited by 1
“Te Mixed Methods Ap-praisal Tool (MMAT) [32] assessed the methodological quality of diferent studies, such as RCTs, cohort studies, case–control studies, and cross-sectional studies.”
Guzman, L. E., Littlewood, A., Creamer, A., & Frank, H. E. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-092378
The effects of Yoga Nidra practice on EEG oscillations: A systematic review
Kachera, V., Patel, D., Mishra, S., et al. (2026) · Advances in Integrative Medicine · doi:10.1016/j.aimed.2026.100683
Hamad, F., Ali, M., Kindawi, M., Mustafa, R., Saeed, A. N. O., Abdelfadeel, W. H. K., & Ibrahim, E. I. H. (2025) · Cureus · doi:10.7759/cureus.86773
“Methodological quality was evaluated using the Newcastle-Ottawa Scale (NOS) [9] for cohort studies and the Mixed Methods Appraisal Tool (MMAT) [10] for non-randomized and mixed-methods studies.”
Bypassing of primary healthcare for emergency departments in Saudi Arabia: A systematic review
Ahmad, M. O. (2025) · Journal of Healthcare Administration · doi:10.33546/joha.3841
Mwila, P. M., & Wambiya, P. O. (2025) · International Journal of Educational Assessment and Evaluation · doi:10.64004/469616d0f8
Kartalova‐O’Doherty, Y., Cannon, M., Sheaf, G., Higgins, A., Keogh, B., Watters, S., Longe, O., Anand-Vembar, S., Staines, L., McEvoy, D., et al. (2025) · HRB Open Research · doi:10.12688/hrbopenres.14184.1
Adlung, C., Paridar, P., Figueroa, C. A., Niehot, C. D., Mandyam, N. K., van Beers, C., & Hinrichs, S. (2025) · medRxiv · doi:10.1101/2025.07.01.25330611
Elliott, S. A., Hartling, L., Ali, S., Lysak, D., Olmstead, D., Knisley, L., Dennett, L., Brooks, H. M., & Scott, S. D. (2026) · JBI Evidence Synthesis · doi:10.1101/2025.07.09.25331190
“The Mixed-Methods Appraisal Tool version 2018 (MMAT) [18] will be used to assess the methodological quality of included studies.”
Challenges and Demands in Indigenous Early Childhood Education: A systematic literature review
Ali, S. A. S., Masnan, A. H., & Ismail, N. K. (2025) · Environment-Behaviour Proceedings Journal · doi:10.21834/e-bpj.v10isi33.7071
Aljuaid, F., Reed, E., Imanpour, S., & Mallinson, D. J. (2025) · JMIR Research Protocols · doi:10.2196/63596
“MMAT is designed to evaluate a broad range of empirical studies, including qualitative, quantitative, and mixed methods designs [16].”
Yimenu, D. K., Quek, H. W., Rafeld, P., Hareem, A., Gebreyohannes, E. A., Wang, K., Seubert, L., Lee, K., & Page, A. T. (2025) · International Journal of Pharmacy Practice · doi:10.1093/ijpp/riaf060 · Cited by 1
Zahedi, S., Tellier, P., Luconi, F., Gore, G. C., & Rodríguez, C. (2025) · Journal of Primary Care & Community Health · doi:10.1177/21501319251356557 · Cited by 1
“33 Interestingly, 28 of the included studies were rated at 80% or higher and 5 studies rated at 60%, and only 1 study rated at 40%.”
Cultural Influences of Alcohol Use Among Latino Farmworkers: A Literature Review
Martinez, A. J. S., Wiese, L. K., & Oliveira, G. C. D. (2025) · International Journal for Human Caring · doi:10.20467/ijhc-2025-0008
Gebre, N. T., Gafer, N., Afolabi, O. A., Nkhoma, K. B., Harding, R., Ayers, N., & Hanlon, C. (2025) · doi:10.2196/preprints.80321
Taxonomy of chronic illness research recruitment: a restricted scoping review
Austin, R. C., Karlsen, B., Richardson, A. M., Elwyn, G. J., Storm, M., Husebø, A. M. L., & Urstad, K. H. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13115-8 · Cited by 2
“Quality Assessment: was conducted using the Mixed Methods Appraisal Tool (MMAT) [31].”
Tufă, L. (2025) · International perspectives on aging · doi:10.1007/978-3-031-94565-6_7 · Cited by 3
A Systematic Review of Policies Promoting Physical Activity in Children and Adolescents Worldwide
Mostaghim, S., Jafarı, M., & Teymourlouy, A. A. (2025) · Health Scope · doi:10.5812/healthscope-159315 · Cited by 1
Experiences of patients with cystic fibrosis involved in service development: a scoping review
Livingstone, J. (2025) · British Journal of Healthcare Management · doi:10.12968/bjhc.2024.0082
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.