The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,601–1,700 of 4,578
Edwards, G., Spooner, M., Arnett, R., Kelly, H., Carr, J. C. A., & Illing, J. C. (2025) · Medical Education · doi:10.1111/medu.15660 · Cited by 12
Sam, I., Dang, W., Iu, N., et al. (2025) · BMC Cancer · doi:10.1186/s12885-025-13980-w · Cited by 9
“The Mixed Methods Appraisal Tool (MMAT) is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews [29].”
Mekuriaw, B., Cutler, N. A., & River, J. (2025) · International Journal of Mental Health Nursing · doi:10.1111/inm.70035 · Cited by 15
“Since this review included qualitative, quantitative, and mixed method studies, we used the Mixed Method Appraisal Tool (MMAT; Hong et al. 2018) to evaluate the quality of each article.”
Fazli, Z., Sadeghi, M., Vali, M., et al. (2025) · Environmental Health · doi:10.1186/s12940-025-01186-3 · Cited by 9
“In Part II of this document, indicators are added for some criteria [15].”
Ståhl, C., Lundqvist, D., & Reineholm, C. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106917 · Cited by 16
Zakhmi, K., Ushmani, A., Mohanty, M. R., Agrawal, S., Banduni, A., & Rao, S. S. K. (2025) · Cureus · doi:10.7759/cureus.85446 · Cited by 22
“To evaluate the methodological quality of the included studies, the Mixed Methods Appraisal Tool (MMAT) 2018 version [10] was employed.”
Nguyen, L., Walters, J., Hutchinson, E., Liu, Y., Li, X., & Gudmundsson, C. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.119683 · Cited by 16
Nguyen, L., Walters, J., Spies, B., Coppus, A., Massie, J., & Bertonha, T. P. (2025) · Food Frontiers · doi:10.1002/fft2.70064 · Cited by 21
School Success for Migrant and Refugee Children: A Systematic Literature Review
Aiello, E., Chiappelli, T., Grigoli, A. R. D., Mancaniello, M. R., Mara, L. C., & Sordé, T. (2025) · SAGE Open · doi:10.1177/21582440251330126 · Cited by 10
“Additionally, the quality of the empirical articles included in the sample was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Psychological safety in software workplaces: A systematic literature review
de Santana, B. S., Monte, L., Silva, B. S. D. A., Carneiro, G. D. F., Freire, S., Santos, J. A. M., & Mendonça, M. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107838 · Cited by 27
“We assessed the methodological quality of the included studies using the Mixed Methods Appraisal Tool (MMAT, 2018 version) Hong et al. (2018).”
Walters, J., Nguyen, L., Liu, Y., Ijurco, S. M., Evans, S., Chacos, N., Duran, M., & Smith, C. (2025) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2025.102491 · Cited by 14
Badawy, W., & Shaban, M. (2025) · Teaching and learning in nursing · doi:10.1016/j.teln.2025.08.003 · Cited by 16
Röscher, P., & Wyk, J. V. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01473-9 · Cited by 9
“The overall quality for each included study will be calculated according to the following MMAT guidelines (score = number of criteria met/total score in each domain).”
BackgroundMethodInfluentialWhat Do We Know About Teamwork in Chinese Hospitals? A Systematic Review
Wang, H., Buljac‐Samardžić, M., Wang, W., van Wijngaarden, J. D. H., Yuan, S., & van de Klundert, J. J. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.735754 · Cited by 12
“Quality Assessment The Mixed Methods Appraisal Tool (MMAT) was used to assess the methodological quality of the included studies (24).”
MethodExploring the life contexts of patrolling police officers in the European Union – A scoping review
Valmari, E. G., Ghazinour, M., Nygren, U., & Gilenstam, K. (2022) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2022.2041088 · Cited by 13
“The Mixed Method Appraisal Tool (MMAT) [64] was used, which evaluates studies using qualitative, quantitative randomized controlled, quantitative non-randomised, quantitative descriptive, and mixedmethods study designs.”
MethodSong, T., Yu, P., & Zhang, Z. (2022) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/3511616.3513110 · Cited by 10
“Non-RCT studies were appraised using the five criteria in Section 3 of the MMAT [32], checking whether (1) the participants could represent the target population; (2) the measurements were appropriate for the outcome and intervention; (3) no less than 80% of the outcomes were reported; (4) confounding factors were con…”
MethodO'Mahony, B., Kerins, C., Murrin, C. M., & Kelly, C. N. M. (2020) · HRB Open Research · doi:10.12688/hrbopenres.13041.1 · Cited by 8
Roberts, C., Toohey, K. L., & Paterson, C. C. (2022) · Cancer Nursing · doi:10.1097/ncc.0000000000001172 · Cited by 9
Suicidal Behaviors and Risk Taking Among Homeless Individuals: A Systematic Review and Meta-Analysis
Armoon, B., Griffiths, M. D., Mohammadi, R., & Ahounbar, E. (2023) · Community Mental Health Journal · doi:10.1007/s10597-023-01111-3 · Cited by 14
Communities of practice in residential aged care: A rapid review
Read, M., Peters, S., Bennett, N. J., Francis, J., Fetherstonhaugh, D., Lim, W. K., & Tropea, J. (2023) · International Journal of Older People Nursing · doi:10.1111/opn.12563 · Cited by 10
Medway, P., Hutchinson, A. M., & Sweet, L. P. (2023) · Sexual & Reproductive Healthcare · doi:10.1016/j.srhc.2023.100900 · Cited by 9
Disease activity in primary progressive multiple sclerosis: a systematic review and meta-analysis
Blok, K. M., van Rosmalen, J., Tebayna, N., Smolders, J., Wokke, B. H. A., & de Beukelaar, J. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1277477 · Cited by 12
“Because a wide variety of study designs could be included, methodological quality was assessed using the Mixed Methods Appraisal Tool (MMAT) (18).”
MethodThe role of anticipated affect in the context of physical activity: a scoping review
Feil, K., Fritsch, J., Weyland, S., Warmbrunn, U., & Jekauc, D. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2023.2275249 · Cited by 13
“The quality of the included studies was assessed with the Mixed Methods Appraisal Tool (MMAT, Hong et al., 2018) which provides di ff erent tools for di ff erent study types.”
Gilroy, J., Henningham, M., Meehan, D. E., et al. (2024) · BMC Public Health · doi:10.1186/s12889-024-18070-3 · Cited by 9
“The appraisal was conducted by three researchers (JG, MH, FN) using the Mixed Method Appraisal Tool (MMAT) (version 2018) due to its ability to appraise multiple methodologies [18].”
MethodKocur, W., McLeod, J., Bloch, S., MacDonald, J., Woodward, C., McInnes-Dean, A., Gibbs, J., Saunders, J., BLANDFORD, A. E., Estcourt, C. S., et al. (2024) · Sexual Health · doi:10.1071/sh23168 · Cited by 12
Patil, D. S., Yadav, U. N., George, S., Helbich, M., Ettema, D., & Bailey, A. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01456-w · Cited by 10
Bullen, K., Hall, N. J., Sherwood, J., et al. (2020) · Integrated Healthcare Journal · doi:10.1136/ihj-2019-000026 · Cited by 9
Prospect of low-cost energy conservation in residential energy consumption
Tyagi, R. K., & Vishwakarma, S. (2021) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijpeds.v12.i4.pp2403-2413 · Cited by 9
“Quality of the articles under review; mixed methods appraisal tool (MMAT) is a reliable content evaluation tool [20].”
MethodSalifu, R. S., Hlongwa, M., & Hlongwana, K. W. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047342 · Cited by 12
Enabling women to access preferred methods of contraception: a rapid review and behavioural analysis
Ayorinde, A., Boardman, F., McGranahan, M., et al. (2021) · BMC Public Health · doi:10.1186/s12889-021-12212-7 · Cited by 10
“Quality assessment We used the Mixed Methods Appraisal Tool (MMAT) for quality assessment of studies [11].”
MethodSelf-citationProcess and Outcome Evaluations of Smartphone Apps for Bipolar Disorder: Scoping Review
Tatham, I., Clarke, E., Grieve, K. A., Kaushal, P., Smeddinck, J. D., Barron, E., & Sharma, A. (2021) · Journal of Medical Internet Research · doi:10.2196/29114 · Cited by 10
“The results of their respective compliance with the standards set out in the MMAT can be found in Multimedia Appendix 3.”
MethodInfluentialHirt, J., Langer, G., Wilde, F., Bauernschmidt, D., Meyer, G., & Bieber, A. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-054157 · Cited by 9
Interventions to improve immigrant women's mental health: A systematic review
Luo, Y., Ebina, Y., Kagamiyama, H., & Sato, Y. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16378 · Cited by 12
Hewitt, J. E., Latimer, S. L., Deakin, J. L., Ranse, K., Lawson, C. G. R., & Grealish, L. A. (2022) · Australian journal of advanced nursing · doi:10.37464/2020.392.280 · Cited by 8
How does cloud computing improve cancer information management? A systematic review
Erfannia, L., & Alipour, J. (2022) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2022.101095 · Cited by 15
Cozma, M.-A., Dobrică, E.-C., Shah, P. C., Shellah, D., Găman, M., & Diaconu, C. C. (2022) · Healthcare · doi:10.3390/healthcare10112196 · Cited by 9
“We evaluated the methodological quality and the risk of bias of the analyzed studies using the Methodological Index for Non-Randomized Observational Studies (MINORS) and the Mixed Methods Appraisal Tool (MMAT), respectively [25,26].”
MethodTrends and ethical issues in nursing during disasters: A systematic review
Fithriyyah, Y. N., Alda, A. K., & Haryani, H. (2023) · Nursing Ethics · doi:10.1177/09697330231155602 · Cited by 16
The impact of nurse prescribing on health care delivery for patients with diabetes: a rapid review
Short, K., Andrew, C., Yang, W., & Jamieson, I. (2024) · Journal of Primary Health Care · doi:10.1071/hc23121 · Cited by 11
“The Mixed Methods Appraisal Tool Version 2018 21 was used for appropriate studies.”
MethodAust, A., Walton‐Sonda, D., & Paterson, C. C. (2022) · Seminars in Oncology Nursing · doi:10.1016/j.soncn.2022.151258 · Cited by 9
Lima, E. M. A., Esplen, M. J., Martins, F. F., et al. (2023) · Psycho-Oncology · doi:10.1002/pon.6141 · Cited by 8
Lim, R. B. T., Hoe, K. W. B., Tan, C. G. L., & Zheng, H. (2023) · Evaluation & the Health Professions · doi:10.1177/01632787231188182 · Cited by 8
“Methodological quality of the included studies was independently evaluated by the first two authors using the Mixed Methods Appraisal Tool, MMAT (Hong et al., 2018) as the review contained qualitative, quantitative, and mixed methods Lim et al.”
BackgroundMethodInfluentialKim, G. S., Kim, N., Shim, M., Lee, J. J., & Park, M. K. (2023) · Health & Social Care in the Community · doi:10.1155/2023/8564397 · Cited by 23
“Te Mixed Methods Appraisal Tool (MMAT) version 2018 [23] was used for quality assessment.”
MethodParsons, J., Salman, B., Leach, H., Watson, E., & Atherton, H. (2023) · BJGP Open · doi:10.3399/bjgpo.2023.0110 · Cited by 8
Predicting Adherence to Behavior Change Support Systems Using Machine Learning: Systematic Review
Ekpezu, A. O., Wiafe, I., & B, H. O.-K. A. (2023) · JMIR AI · doi:10.2196/46779 · Cited by 7
“Since the study selection was not limited to a specific study design, the Mixed Methods Appraisal Tool by [26] was used to assess the methodological quality of the downloaded papers (see Multimedia Appendix 2 [13,14,26-35]).”
MethodBarriers to optimal management of cancer pain in sub-Saharan Africa: a scoping review
Magboh, V. O., Saba, O. A., Krause, R., & Forget, P. (2023) · ecancermedicalscience · doi:10.3332/ecancer.2023.1650 · Cited by 9
“To assess the quality of the studies, the mixed methods appraisal tool (MMAT) version 2018 created by Hong et al [17] was utilised.”
MethodHe, H., Li, J., Zhao, L., Lu, H. S., Lu, J., Quan, Y., & Xiu, Z. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17057 · Cited by 9
Howell, J., Deane-King, J., & Maguire, R. (2024) · Journal of Homosexuality · doi:10.1080/00918369.2024.2326891 · Cited by 9
“While higher levels of stigma were reported in certain cities such as New York in one study (Franks et al., 2018), the southern USA was reported as an area in which stigma was higher in a number of studies.”
InfluentialThe role of community pharmacy in the promotion of continence care: A systematic review
Uren, A. D., Dawson, S., Cotterill, N., et al. (2024) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2024.04.010 · Cited by 8
Structured interactions between nurses and patients through intentional rounding: A scoping review
Al‐Anati, A., Molloy, L., Sim, J., et al. (2024) · International Nursing Review · doi:10.1111/inr.12984 · Cited by 8
“MMAT was selected for its validity, reliability, and applicability to qualitative, quantitative, and mixed-methods research (Hong et al., 2018).”
Hamiduzzaman, M., Miles, S., Crook, S., Grove, L., Hewitt, J., Barraclough, F. L., Hawkins, P., Campbell, E., Buster, N., Thomson, K. E., et al. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05632-7 · Cited by 11
“This review adhered to the five steps of an integrative review process as its foundation, established by Whitte-more and Knafl in 2005 [36].”
MethodVillar-García, M., & Pereira‐Moliner, J. (2024) · Cuadernos de Turismo · doi:10.6018/turismo.616421 · Cited by 11
“La metodología cualitativa ayuda a crear teoría gracias a su enfoque inductivo (Hong et al., 2018; Molina-Azorín et al., 2024).”
BackgroundColman, M. D., Millar, J. F. A., Patil, B., Finnegan, D. J., Russell, A., Higson‐Sweeney, N., Aguiar, M. D. S., & Fraser, D. S. (2024) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2024.100893 · Cited by 10
“Only one study investigated the use of AR (García-Batista et al., 2021), the other five focusing on VR. Assessment with the MMAT found all studies in this area were rated 4/5 or above, indicating a high quality of research – a common flaw being that attempts to account for confounding variables such as comorbidity or…”
InfluentialSpanos, S., Dammery, G., Pagano, L., Ellis, L. A., Fisher, G., Smith, C. L., Foo, D., & Braithwaite, J. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11295-3 · Cited by 13
“As the included articles were broad in their publication design, three appraisal tools were used: 1) Mixed-Methods Appraisal Tool (MMAT) [27]; 2) Scale for the quality Assessment of Narrative Reviews (SANRA) [28]; and 3) Joanna Briggs Institute (JBI) Text and Opinion [29].”
MethodAsubiaro, J. (2024) · Cureus · doi:10.7759/cureus.65366 · Cited by 22
“The quality of all included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [29].”
MethodRouleaux, M., Peters-Scheffer, N. C., Lindauer, R., Mevissen, L., & Didden, R. (2024) · Journal of Intellectual Disability Research · doi:10.1111/jir.13178 · Cited by 12
Tower, M., Muir, R., Zimmerman, P.-A. P., Carter, A. G., Maude, R., Hollaway, K., Geia, L. K., Massey, D. L., & Elder, E. G. (2024) · Nurse Education Today · doi:10.1016/j.nedt.2024.106352 · Cited by 13
Integrating Digital Assistive Technologies Into Care Processes: Mixed Methods Study
Hofstetter, S., Zilezinski, M., Behr, D., Kraft, B., Buhtz, C., Paulicke, D., Wolf, A. A., Klus, C., Stoevesandt, D., Schwarz, K., et al. (2024) · JMIR Medical Education · doi:10.2196/54083 · Cited by 8
“The quality criteria of the survey were discussed using the Mixed Methods Appraisal Tool [50].”
MethodWilliams, N., & Brömdal, A. (2024) · International Journal of Transgender Health · doi:10.1080/26895269.2024.2387662 · Cited by 11
Self-Report of Restricted Repetitive Behaviors in Autistic Adults: A Systematic Review
Collis, E., Dark, E., Russell, A., & Brosnan, M. (2024) · Autism in Adulthood · doi:10.1089/aut.2023.0111 · Cited by 10
“The MMAT recommends giving detailed summaries of ratings rather than summed quality scores.”
InfluentialMenon, S., Katona, C., & Glover, N. (2024) · Mental Health Science · doi:10.1002/mhs2.85 · Cited by 12
Thomson, G., & Nowland, R. A. (2024) · Midwifery · doi:10.1016/j.midw.2024.104185 · Cited by 12
Caffeine Placebo Effect in Sport and Exercise: A Systematic Review
Vega-Muñoz, A., Contreras-Barraza, N., Salazar-Sepúlveda, G., Raby, N. D. L., Gil-Marín, M., & Muñoz-Urtubia, N. (2024) · Nutrients · doi:10.3390/nu16183219 · Cited by 17
AlRawashdeh, M. M., Ishak, A., Al-Bunnia, A., et al. (2024) · Pathogens · doi:10.3390/pathogens13090817 · Cited by 8
“Finally, for mixed-methods studies, the Mixed Methods Appraisal Tool version 2018 was used [18].”
MethodLuigi, M., Martinez, L.-A., Roy, L., & Crocker, A. G. (2024) · Aggression and Violent Behavior · doi:10.1016/j.avb.2024.102009 · Cited by 13
Dilworth, S., Doherty, E. L., Mallise, C. A., Licata, M., Hollis, J., Wynne, O., Lane, C., Wolfenden, L., Wiggers, J., & Kingsland, M. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00655-z · Cited by 12
“The Mixed Methods Appraisal Tool (MMAT) [41, 42] was used to appraise the methodological quality of included studies.”
MethodDeasy, E., Seoighe, A., Ryan, C. A., Byrne, S., & Dalton, K. (2024) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2024.100522 · Cited by 16
Munawar, K., Mukhtar, F., Roy, M., Majeed, N., & Jalaludin, M. Y. (2024) · Psychology Health & Medicine · doi:10.1080/13548506.2024.2421461 · Cited by 43
Hamiduzzaman, M., Gaffney, H. J., Jindal, S., Patra, M., Gudur, R. R., Pit, S., & Rahman, M. A. (2024) · Journal of Applied Gerontology · doi:10.1177/07334648241296791 · Cited by 10
“The MMAT scale spanned from 0, indicating no criteria met, to 5, indicating all criteria met, as detailed by Hong et al. in 2018, was used to assign numerical scores to the articles (Hong et al., 2018).”
Crawford, J., Kelder, J., & Wilson, G. (2024) · Quality & Quantity · doi:10.1007/s11135-024-02003-8 · Cited by 12
“…subsequently developed include a quality assessment with diverse studies tool (QuADS) (Harrison et al. 2021 ); a mixed methods appraisal tool (MMAT: Hong et al. 2018 ); and six qualitative synthesis tools suitable for the quality appraisal of qualitative research studies identified by Majid and…”
Fang, W., Phung, H. N., Olley, R., & Lee, P. (2024) · Healthcare · doi:10.3390/healthcare12232468 · Cited by 8
Akbayrak, E., Powell, P. A., Tunc, N., & Barnes, S. (2024) · The Gerontologist · doi:10.1093/geront/gnae176 · Cited by 11
“Responses were categorized as “can’t tell,” “no,” or “yes” (Hong et al., 2018).”
STEM service learning in higher education: A systematic literature review
Mahmud, S. N. D., & Ismail, N. K. (2024) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/15705 · Cited by 28
“(2018) created the mixed-method appraisal tool (MMAT), which enables researchers to evaluate a systematic mixed-methods review and encompasses five distinct types of studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and…”
Interventions used to reduce infectious aerosol concentrations in hospitals—a review
Brady, G., Bennin, F., de Koning, R., et al. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2024.102990 · Cited by 8
Alcohol use among populations with autism spectrum disorder: narrative systematic review
Barber, W., Aslan, B., Meynen, T., Marsden, J., Chamberlain, S. R., Paleri, V., & Sinclair, J. (2025) · BJPsych Open · doi:10.1192/bjo.2024.824 · Cited by 12
“Each study was evaluated by design and rated across five criteria, individual to each study design (yes, 1; no, 0), with an overall quality score calculated as a percentage of total criteria met (presented in Table 1 60 ).”
Dapari, R., Jumidey, A. Q., Manaf, R. A., et al. (2025) · Discover Social Science and Health · doi:10.1007/s44155-025-00181-w · Cited by 8
“Quality appraisal of each article was conducted using the Mixed Method Appraisal Tool (MMAT) 2018 [16].”
Effect of Fluoride on Gut Microbiota: A Systematic Review
Yasin, M., Zohoori, F. V., Kumah, E. A., Subramanian, M., Dean, P., & Orr, C. H. (2025) · Nutrition Reviews · doi:10.1093/nutrit/nuae202 · Cited by 8
“30 The Mixed Methods Appraisal Tool contains 5 and 6 evaluation elements/questions each for the various study designs, with a judgement of “yes,” “no,” and “unclear” for each question.”
Morin, C., & Alfahl, Z. (2025) · Journal of Applied Microbiology · doi:10.1093/jambio/lxaf066 · Cited by 8
Chivasa, T., Dhlamini, M., Maviza, A., Nunu, W. N., & Tsoka-Gwegweni, J. M. (2025) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph22040591 · Cited by 9
“MMAT is a highly suitable quality evaluation tool for a scoping review of the literature owing to its versatility and comprehensive criteria for different study designs [46].”
Mbatha, T., van Rensburg, N. J., & Morris-Eyton, H. (2025) · International Journal of Sports Science & Coaching · doi:10.1177/17479541251333931 · Cited by 21
“Using the Mixed Method Appraisal Tool (MMAT, 32 ), two authors independently appraised the methodological quality of the included studies.”
Harkins, S., Hazi, A. K., Guglielminotti, J. R., Landau, R., & Barcelona, V. (2025) · International Journal of Obstetric Anesthesia · doi:10.1016/j.ijoa.2025.104379 · Cited by 12
Impacts of Breaching Planetary Boundaries on Human Health: Current and Future Threats
Kemarau, R. A., Sakawi, Z., Eboy, O. V., Jaafar, W. S. W. M., Sa’adi, Z., & Suab, S. A. (2025) · GeoHealth · doi:10.1029/2024gh001107 · Cited by 14
“We adopted a qualitative analysis approach as proposed by Petticrew and Roberts (2006), integrating quality assessment criteria from Hong et al. (2018).”
Whitehall, J. M., Cook, E., Všetečková, J., Jones, K., Pappas, Y., Donald, L., & Chater, A. (2025) · International Journal of Obesity · doi:10.1038/s41366-025-01811-8 · Cited by 12
“An overall numerical MMAT-V 2018 score for each study was not calculated as the score does not highlight the problematic areas of the study in terms of quality [41]; instead, a more in-depth explanation of each speci fi c criterion has been reported [40] (see Table 1).”
Adaranijo, E. T., Marshall, C. R., Ong, A., & Nwachukwu, B. (2025) · Health & Social Work · doi:10.1093/hsw/hlaf014 · Cited by 13
Osei‐Safo, E. K., McIntosh, J., Onwuka, S. R., Torkel, S., McGowan, M., Cocotis, K., Angel, C., Varatharaj, S., Teede, H. J., Melder, A., et al. (2025) · Obesity Reviews · doi:10.1111/obr.13967 · Cited by 9
“The following quality appraisal tools were used: Critical Appraisal Skills Program (CASP) Qualitative Studies Checklist [24], Center for Evidence-Based Management (CEMB) Critical Appraisal Checklist for a Cross-Sectional Study [25], and the Mixed Methods Appraisal Tool (MMAT) [26].”
Shaleha, R. R. A., & Roque, N. A. (2025) · Digital Health · doi:10.1177/20552076251343989 · Cited by 8
“12 We used it to guide our evaluation.”
Butler, H., Gott, M., & Robinson, J. (2025) · International Journal of Mental Health · doi:10.1080/00207411.2025.2526215 · Cited by 13
“These studies were appraised using the mixed methods appraisal too (MMAT) version 2018 (Hong et al., 2018).”
Sun, X., Qu, Z., Zhang, X., Zhang, Y., Zhang, X., Zhao, H., & Zhang, H. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2025.1626309 · Cited by 23
Crestani, M., Cook, C. E., Leuci, C., Carletto, L., Garzonio, F., Palese, A., Chester, R., Baets, L. D., Gandolfi, M., Rossettini, G., et al. (2025) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2025.13432 · Cited by 11
Ounsaneha, W., Bhatta, J., & Rattanapan, C. (2025) · Green Technologies and Sustainability · doi:10.1016/j.grets.2025.100256 · Cited by 16
Johnson, E., Seed, L. M., Biju, A., & Tulinius, C. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-103159 · Cited by 9
“The protocol initially specified the use of Critical Appraisal Skills Programme (CASP) check-lists; 16 however, five of the included were cross-sectional, for which there is not an appropriate CASP checklist— MMAT was therefore used.”
Management of Preoperative Anxiety via Virtual Reality Technology: A Systematic Review
Alimonaki, E. C., Bothou, Α., Diamanti, A., Δελτσίδου, Ά., Paliatsiou, S., Karampas, G., & Kyrkou, G. A. (2025) · Nursing Reports · doi:10.3390/nursrep15080268 · Cited by 17
“The mixed method appraisal tool (MMAT; Hong et al., 2018) is a critical appraisal instrument developed for the evaluation stage of systematic reviews [22].”
Morillon, G. F., Guillon, M., Laberge, M., Sowanou, A., & Poder, T. G. (2025) · BMC Family Practice · doi:10.1186/s12875-025-02931-w · Cited by 14
“As we considered qualitative, quantitative, and mixed method studies, we used the second version of the Mixed Methods Appraisal Tool (MMAT) to assess the quality of studies [18].”
Ahmad, R., Maulud, K. N. A., Zamir, U. B., Razali, S. F. M., Yaseen, Z. M., Pradhan, B., Khan, M. N., & Eshquvvatov, B. B. (2025) · Geomatics Natural Hazards and Risk · doi:10.1080/19475705.2025.2549487 · Cited by 18
“The criteria used for the evaluation of selected articles (Hong et al. 2018).”
Kossinna, C., Lau, C. L., Mayr, H. L., Wilkinson, S. A., Young, A., Meloncelli, N. J. L., Cameron, A. E., Rigby, R. R., & Hickman, I. J. (2025) · Nutrition in Clinical Practice · doi:10.1002/ncp.70016 · Cited by 15
“20 Each study was assessed by two independent reviewers (C.K. and C.L.L.), and a third independent assessor (R.R.) reviewed all articles and resolved discrepancies through consensus discussions.”
Karbasi, A. B., Iloegbu, C., Ruan, C., OSEI-TUTU, N. H., Patel, K. H., Frerichs, L., Patena, J. V., Vieira, D. L., Adenikinju, D., Samuels, L., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0328515 · Cited by 12
“Mixed Method studies were evaluated based on the Mixed Methods Appraisal Tool (MMAT) tool [48].”
Gauch, M., Spelter, B., Geschke, K., Köb, A., Tüscher, O., Heinrich, I., & Corsten, S. (2025) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.70129 · Cited by 7
Mancinelli, E., & Filippi, B. (2025) · EClinicalMedicine · doi:10.1016/j.eclinm.2025.103512 · Cited by 24
“Evidence for the Encoding and Interpreting phase (Table 2) was found in 34 studies, retrieving barriers and facilitators from 33.”
Child engagement during interaction with digital and robotic activities: a systematic review
Margheri, V., Martucci, A., Bei, E., Graziani, D., Scatigna, S., Guazzini, A., & Pecini, C. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1568028 · Cited by 13
ARSLANLI, S. E., Duru, H. A., Ünal, E., et al. (2025) · BMC Nursing · doi:10.1186/s12912-025-03520-z · Cited by 8
“The MMAT is a quality appraisal tool designed for literature that includes quantitative, qualitative, and mixed meth-ods research articles [32].”
Mental health and well‐being in esports: A scoping review
Poulus, D., Nicholson, M., Obine, E. A. C., Drew, M., FISHER, K., Vella, S. A., Seidler, Z. E., & Sharpe, B. T. (2026) · Applied Psychology · doi:10.1111/apps.70055 · Cited by 18
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.