The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,001–1,100 of 4,578
Association between adult-onset hearing loss and dementia biomarkers: A systematic review
Lasica, A. B., Sheppard, J., Yu, R., Livingston, G., Ridgway, N., Omar, R., Schilder, A. G., & Costafreda, S. G. (2024) · Ageing Research Reviews · doi:10.1016/j.arr.2024.102647 · Cited by 20
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT), version 2018 (Hong et al., 2018).”
Du, M., Hancox, J. E., Hooper, O., Sandford, R. A., & Huang, C. (2025) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2025.2471759 · Cited by 21
“For this scoping review, two reviewers (first and fifth authors) independently assessed the quality of each included study using the MMAT.”
InfluentialArnold, L., Aryal, S., Hong, B., Nitharsan, M., Shah, A., Ahmed, W., Lilani, Z., Su, W., & Piaggio, D. (2025) · Applied Sciences · doi:10.3390/app15063308 · Cited by 20
Paes, V. M., Ting, A., Masters, J., et al. (2025) · The Bone & Joint Journal · doi:10.1302/0301-620x.107b6.bjj-2024-1255.r1 · Cited by 17
Patel, S. B., Iqbal, F. M., Lam, K., Acharya, A., Ashrafian, H., & Darzi, A. W. (2025) · Journal of Medical Internet Research · doi:10.2196/56711 · Cited by 35
“Risk of bias was assessed using the Mixed Methods Appraisal Tool (MMAT) [15].”
McGrath, M., Bagul, D., & du Toit, S. H. J. (2021) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2021.1898675 · Cited by 17
“[15] no studies were excluded based on the appraisal and the findings of the appraisal are presented in full so that readers can make informed judgments about the quality of included papers.”
BackgroundMethodInfluentialCasey, T. W., Mason, H. M., Huang, J., & Franklin, R. C. (2021) · Safety · doi:10.3390/safety7040076 · Cited by 25
“It is discouraged to calculate an overall score using the MMAT and the CASP [18,19].”
MethodCalonge, D. S., Connor, M., Hultberg, P. T., Shah, M. A., & Aguerrebere, P. M. (2021) · Journal of Educational Studies and Multidisciplinary Approaches · doi:10.51383/jesma.2022.22 · Cited by 22
Shin, J., Konlan, K. D., & Mensah, E. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260411 · Cited by 23
“Two researchers independently appraised the quality of each study as suggested by Hong et al., 2018 [19].”
MethodSecondary Trauma and Related Concepts in Psychologists: A Systematic Review
Pellegrini, S., Moore, P., & Murphy, M. F. (2022) · Journal of Aggression Maltreatment & Trauma · doi:10.1080/10926771.2021.2019156 · Cited by 29
Grant, S., Hartanto, S., Sivasubramaniam, D., & Heritage, K. (2022) · Health & Social Care in the Community · doi:10.1111/hsc.14039 · Cited by 18
“This tool includes five methodological criteria questions (yes, no and unclear) for each study type (see Mixed Methods Appraisal Tool 2018 user guide; Hong et al., 2018) and is considered an effective and efficient quality appraisal tool for empirical research, with good inter- rater reliability (Pace et al., 2012).”
MethodInfluentialRibeiro, C., Tsang, L., Lin, B., Kemp‐Smith, K., Phillips, J., & Furness, J. W. (2022) · Physiotherapy Theory and Practice · doi:10.1080/09593985.2022.2122915 · Cited by 17
“(Appendix 2, Table 2e) (Hong et al., 2018).”
BackgroundMethodInfluentialVillegas, F. E., van der Steen, S., & Minnaert, A. E. M. G. (2022) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-022-00346-2 · Cited by 30
Fuentes, M. J. F. D. O., Keegan, B. J., Jones, M. V., & Macintyre, T. E. (2022) · Urban forestry & urban greening · doi:10.1016/j.ufug.2022.127796 · Cited by 32
Park, S. Y., Kim, Y., Kim, S., & Katapodi, M. C. (2022) · Genetics in Medicine · doi:10.1016/j.gim.2022.100001 · Cited by 20
Radež, J., Waite, F., Izon, E., & Johns, L. (2023) · Early Intervention in Psychiatry · doi:10.1111/eip.13365 · Cited by 22
“The MMAT permits the reviewer to appraise the quality of five categories of studies – qualitative, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies (Hong et al., 2018).”
MethodInfluentialSantos, L., Miguel, R. R., Pinheiro, M. D. R., & Rijo, D. (2023) · Children and Youth Services Review · doi:10.1016/j.childyouth.2023.106839 · Cited by 26
Longo, E., Monteiro, R. F. L., Hidalgo‐Robles, Á., Paleg, G., Shrader, C., & de Campos, A. C. (2023) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2023.1110552 · Cited by 17
Tsiloni, E., Dragioti, E., Gouva, M., Vassilopoulos, S. P., & Mentis, M. N. (2023) · Gerontology & Geriatrics Education · doi:10.1080/02701960.2023.2189248 · Cited by 33
“MMAT is a critical assessment tool used to assess the quality of five categories of studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies (MMAT, Hong et al., 2018).”
MethodHorgan, S., Saab, M. M., Drennan, J., Keane, D., & Hegarty, J. M. (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2023.103637 · Cited by 27
Buuringen, M. V. D. V., der Sar, R. H.-V., Verbeek, H., Achterberg, W. P., & Caljouw, M. A. A. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1044271 · Cited by 26
“(24) was used to assess the methodological quality of all included individual studies.”
MethodCounseling interventions for victims of intimate partner violence: A systematic review
Craven, L. C., Fields, A. M., Carlson, R. G., Combs, E. M., & Howe, E. S. (2023) · Journal of Counseling & Development · doi:10.1002/jcad.12478 · Cited by 29
Does media content have an impact on help-seeking behaviors for mental illness? A systematic review
Goodwin, J., & Behan, L. (2023) · Mental Health Review Journal · doi:10.1108/mhrj-06-2022-0038 · Cited by 30
Royston, R., Naughton, S., Hassiotis, A., Jahoda, A. J., Ali, A., Chauhan, U., Cooper, S., Kouroupa, A., Steed, L., Strydom, A., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0285590 · Cited by 18
Gillen, P. A., Bamidele, O. O., & Healy, M. (2023) · Midwifery · doi:10.1016/j.midw.2023.103733 · Cited by 26
Scott, H. M., Braybrook, D. E., Harðardóttir, D., Ellis-Smith, C. I., Harding, R., C-POS, O. B. O., Anderson, A., Bayly, J., Bate, L., Bluebond‐Langner, M., et al. (2023) · Health and Quality of Life Outcomes · doi:10.1186/s12955-023-02143-9 · Cited by 20
Kanyemba, R., Govender, K., Dzomba, A., Mashamba-Thompson, T. P., & Mantell, J. E. (2023) · AIDS and Behavior · doi:10.1007/s10461-023-04054-8 · Cited by 20
Disability, equity, and measurements of livability: A scoping review
Bohorquez, N. G., Stafford, L., McPhail, S., Selim, S. M., Kularatna, S., & Malatzky, C. A. R. (2023) · Disability and health journal · doi:10.1016/j.dhjo.2023.101521 · Cited by 18
Albougami, A. S. B. (2023) · Applied Sciences · doi:10.3390/app131810403 · Cited by 18
Bolanis, D., Vergunst, F., Mavoa, S., Schmelefske, E., Khoury, B., Turecki, G., Orri, M., & Geoffroy, M. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.167451 · Cited by 23
Schoors, M. V., Lierde, E. V., Steeman, K., Verhofstadt, L., & Lemmens, G. M. D. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1243784 · Cited by 35
Tuschick, E., Barker, J., Giles, E. L., Jones, S. E., Hogg, J., Kanmodi, K. K., Sill, J., & Sykes, K. (2024) · Psycho-Oncology · doi:10.1002/pon.6274 · Cited by 27
“24 The MMAT was chosen as it has been used in previously published mixed method systematic reviews that focused on barriers and facilitators within cancer contexts.”
BackgroundMethodInfluentialAspinall, C., Slark, J., Parr, J. M., et al. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16043 · Cited by 17
Lázaro, E., Yepez, J.-C., Marín-Maicas, P., López-Masés, P., Gimeno, T., de Paúl, S., & Moscardó, V. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100407 · Cited by 20
Koorts, H., Ma, J., Swain, C. T. V., Rutter, H., Salmon, J., & Bolton, K. A. (2024) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-024-01579-6 · Cited by 25
“The Mixed Methods Appraisal Tool (MMAT) version 2018 [39] was used to appraise the quality of included studies as it encompasses multiple study designs (e.g., qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies)…”
MethodImpacts of suicide bereavement on men: a systematic review
Logan, N., Krysinska, K. E., & Andriessen, K. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1372974 · Cited by 28
“Evaluation of the quality and potential bias in included studies was conducted using the Mixed Methods Appraisal Tool (MMAT) (35).”
MethodInfluentialDiagnostic delay of sarcoidosis: an integrated systematic review
Namsrai, T., Phillips, C. B., Parkinson, A., Gregory, D., Kelly, E., Cook, M., & Desborough, J. (2024) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-024-03152-7 · Cited by 27
“Included studies were quality appraised using the Mixed Methods Appraisal Tool (MMAT) [21].”
MethodPatient and caregiver shared experiences of pulmonary fibrosis (PF): A systematic literature review
Wells, M., Harding, S. A., Dixon, G., Buckley, K., Russell, A., & Barratt, S. L. (2024) · Respiratory Medicine · doi:10.1016/j.rmed.2024.107659 · Cited by 19
Pemovska, T., Loizou, S., Appleton, R., Spain, D., Stefanidou, T., Kular, A., Cooper, R. E., Greenburgh, A., Griffiths, J., Barnett, P., et al. (2024) · Psychological Medicine · doi:10.1017/s0033291724001089 · Cited by 28
“Reviewers assessed study quality using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
MethodTeachers’ views on gameful practices – A scoping review
Bacsa-Károlyi, B., & Fehérvári, A. (2024) · Teaching and Teacher Education · doi:10.1016/j.tate.2024.104730 · Cited by 32
The impact of advertising on women’s self-perception: a systematic review
Dai, Y., Zhu, Z., & Guo, W. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1430079 · Cited by 47
Vo, K., Sarkar, M., White, P. J., & Yuriev, E. (2025) · Journal of Chemical Education · doi:10.1021/acs.jchemed.4c01327 · Cited by 30
Chougui, K., Addab, S., Palomo, T., Morin, S. N., Veilleux, L., Bernstein, M., Thorstad, K., Hamdy, R. C., & Tsimicalis, A. (2020) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.61497 · Cited by 21
“For each study, the methodological quality was evaluated by giving a score ranging from 0% (all five criteria were unmet) to 100% (all five criteria were met; Hong et al., 2018).”
MethodChater, A., Family, H. E., Lim, R. H. M., & Courtenay, M. (2020) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkaa335 · Cited by 16
Kouter, K., & Paska, A. V. (2020) · Epigenomics · doi:10.2217/epi-2020-0196 · Cited by 21
van der Wal, R., Loutfi, D., Hong, Q. N., Vedel, I., Cockcroft, A., Johri, M., & Andersson, N. (2021) · Journal of the International AIDS Society · doi:10.1002/jia2.25787 · Cited by 18
“To assess the robustness of included studies (synthesis step 4), two reviewers (RW and DL) independently appraised included papers with the Mixed Methods Appraisal Tool [36].”
MethodSelf-citationYang, Y., Conti, J., McMaster, C. M., & Hay, P. (2021) · Nutrients · doi:10.3390/nu13124490 · Cited by 29
“Adapted from [28], with permission from authors, 2018.”
BackgroundMethodCooke, E., Henderson‐Wilson, C., Warner, E., & LaMontagne, A. D. (2022) · Health Promotion International · doi:10.1093/heapro/daac001 · Cited by 22
The Mental Health Impact of the COVID-19 Pandemic on Older Adults in China: A Systematic Review
Liu, J., Kwan, C., Deng, J., & Hu, Y. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192114362 · Cited by 20
“The other two investigators (DJ and HYX) independently assessed the methodological quality of mixed-method studies using the Mixed Methods Appraisal Tool [17].”
MethodPaediatric palliative care: a systematic review
Nilsson, S., Öhlén, J., Hessman, E., & Brännström, M. (2019) · BMJ Supportive & Palliative Care · doi:10.1136/bmjspcare-2019-001934 · Cited by 25
Sasso, I., & Sansour, T. (2024) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-024-00474-x · Cited by 28
Anderberg, I., Kemps, E., & Prichard, I. (2025) · Body Image · doi:10.1016/j.bodyim.2024.101836 · Cited by 16
FACILITADORES E BARREIRAS DO ENVOLVIMENTO DO PACIENTE NOS SERVIÇOS HOSPITALARES: REVISÃO INTEGRATIVA
de Souza, A. D. Z., Hoffmeister, L. V., & de Moura, G. M. S. S. (2022) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2020-0395pt · Cited by 16
The influence of social media on student nurses: A systematic mixed-studies review
Tan, M. Y. N., Ni, Z., Liu, A. S. H., & Shorey, S. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.106000 · Cited by 26
van der Mee, F. A. M., Schaper, F., Jansen, J., Bons, J. A. P., Meex, S. J. R., & Cals, J. (2024) · Journal of Medical Internet Research · doi:10.2196/53993 · Cited by 29
“To assess the quality and risk of bias of all included studies, the Mixed Methods Appraisal Tool (MMAT) was used [38].”
Mellado, A., del Río, M. T., Andreucci-Annunziata, P., & Molina, M. E. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1308131 · Cited by 32
Liu, J., Yu, H., Bleakney, A., & Jan, Y. (2024) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2024.1461512 · Cited by 41
Harris, D., Dlima, S. D., Gluchowski, A., Hall, A., Elliott, E., & Munford, L. A. (2024) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-024-01666-8 · Cited by 29
“Two reviewers independently assessed methodological quality of each included study using the Mixed Methods Appraisal Tool (MMAT) version 2018 [35].”
MethodBeyene, L. S., Strand, E. B., Misund, A. R., Barder, H. E., Brente, T. L., & Størksen, H. T. (2024) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2024.100266 · Cited by 34
Cook, M., Pennay, A. E., Caluzzi, G., Cooklin, A., MacLean, S. J., Riordan, B. C., Torney, A., & Callinan, S. (2025) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2025.104763 · Cited by 19
Dimitriadou, I., Eloranta, S., Šteinmiller, J., Saridi, M. I., Lundberg, A., Häger, M., Hjaltadóttir, I., Skúladóttir, S. S., Korsström, N., Mört, S., et al. (2025) · Geriatrics · doi:10.3390/geriatrics10020048 · Cited by 19
“To reduce the risk of bias, upon completion of the literature search, the included studies underwent rigorous evaluation via the MMAT [18].”
Low back pain and sitting time, posture and behavior in office workers: A scoping review
Alaca, N., Acar, A. Ö., & Öztürk, S. (2025) · Journal of Back and Musculoskeletal Rehabilitation · doi:10.1177/10538127251320320 · Cited by 35
“The MMAT quality assessment was also used to identify consistent methodological weaknesses in the existing evidence base and to generate recommendations for improving future studies.”
InfluentialAlva, J. I., Brewster, R. C., Mahmood, Z., Harrell, K. M., Kaiser, N. C., Riesthuis, P., YoungSciortino, K., Brunet, H. E., Johnson, M. E., & Kovach, S. (2025) · The Clinical Neuropsychologist · doi:10.1080/13854046.2025.2493343 · Cited by 19
“The revised Cochrane risk-of-bias tools for randomized crossover trials (RoB 2; Sterne et al., 2019) and the Mixed-Method Appraisal Tool (MMAT; Hong et al., 2018) were not appropriate for our review as many bias domains and associated signaling questions do not apply to the types of studies we…”
Mchunu, G. G., Kuupiel, D., Ncama, B. P., et al. (2025) · BMC Women s Health · doi:10.1186/s12905-025-03821-0 · Cited by 16
“The Mixed Method Quality Appraisal Tool (MMAT) Version 2018 [23] was utilised to assess the methodological quality and potential risk of bias in the included studies.”
Paes, V. M., Ting, A., Masters, J., et al. (2025) · Bone & Joint Open · doi:10.1302/2633-1462.67.bjo-2024-0243.r1 · Cited by 16
Artificial intelligence and robotics in predictive maintenance: a comprehensive review
Azeta, J., Omeche, T. T., Daniyan, I. A., Abiola, J. O., Daniyan, L., Phuluwa, H. S., & Muvunzi, R. (2026) · Frontiers in Mechanical Engineering · doi:10.3389/fmech.2025.1722114 · Cited by 34
Barriers to implementation of emergency obstetric and neonatal care in rural Pakistan
Haider, S., Ali, R. F., Ahmed, M., Humayon, A. A., Sajjad, M., & Ahmad, J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0224161 · Cited by 20
Hallam, B., Rees, J., Petersen, I., Cooper, C., Avgerinou, C., & Walters, K. R. (2021) · Family Practice · doi:10.1093/fampra/cmab014 · Cited by 18
“The use of MMAT in mixed-method reviews has been validated, which then allows quality appraisal for the variety of study designs to be completed using one tool (35,36).”
MethodInfluentialPark, S., & Ahmed, R. (2021) · Health Communication · doi:10.1080/10410236.2021.2010328 · Cited by 31
“The MMAT 2018 version checklist (Hong et al., 2018) was used for quality appraisal.”
MethodPotentially modifiable associates of anxiety in people with multiple sclerosis: a systematic review
Fahy, A., & Maguire, R. (2022) · Disability and Rehabilitation · doi:10.1080/09638288.2021.2022776 · Cited by 19
Pregnancy outcomes in women with active anorexia nervosa: a systematic review
Pan, J. R., Li, T. Y., Tucker, D., & Chen, K. Y. (2022) · Journal of Eating Disorders · doi:10.1186/s40337-022-00551-8 · Cited by 23
“The MMAT is a set of five, five-point checklists, each corresponding to a different study design [22].”
MethodCoordination Models for Cancer Care in Low- and Middle-Income Countries: A Scoping Review
Lubuzo, B., Hlongwana, K. W., Hlongwa, M., & Ginindza, T. G. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19137906 · Cited by 21
“We used the mixed method appraisal tool (MMAT) version 2018 to evaluate the quality of the included articles [53].”
MethodInfluentialAllum, L., Apps, C., Pattison, N. A., Connolly, B. A., & Rose, L. (2022) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2022.103302 · Cited by 17
Leave events among Aboriginal and Torres Strait Islander people: a systematic review
Coombes, J., Hunter, K., Bennett‐Brook, K., Porykali, B., Ryder, C., Banks, M., Egana, N., Mackean, T., Sazali, S., Bourke, E., et al. (2022) · BMC Public Health · doi:10.1186/s12889-022-13896-1 · Cited by 20
“Quality assessment The Mixed Methods Appraisal tool (MMAT) was used to assess the quality of peer reviewed studies [15].”
MethodSaab, M. M., McCarthy, M. J., O’Driscoll, M., Sahm, L. J., Leahy‐Warren, P., Noonan, B. J., Fitzgerald, S. M., O’Malley, M., Lyons, N., Burns, H. E., et al. (2022) · npj Primary Care Respiratory Medicine · doi:10.1038/s41533-022-00312-9 · Cited by 22
Chua, C. M. S., Nantsupawat, A., Wichaikhum, O., & Shorey, S. (2022) · Nurse Education Today · doi:10.1016/j.nedt.2022.105621 · Cited by 17
Wei, W., & Balser, S. (2022) · Trauma Violence & Abuse · doi:10.1177/15248380221140123 · Cited by 16
Li, Z., Beck, E., McIlfatrick, S., & Hasson, F. (2023) · Journal of Palliative Care · doi:10.1177/08258597231158321 · Cited by 16
“Appraisals by MMAT were presented with detailed information on each criterion rather than the total score of the MMAT for each study guided by the tool’s developer.(25) Appraisals by Joanna Briggs Institute Critical Appraisal Tools were presented and classified as High (a score below 49%), Moderate (50-74%), and Low (…”
MethodBounsall, K., Milne-Ives, M., Hall, A., Carroll, C., & Meinert, E. (2023) · JMIR Research Protocols · doi:10.2196/46581 · Cited by 16
“For nonrandomized studies, the Mixed Methods Appraisal Tool will be used [58] due to its appraisal of most common study types and designs.”
MethodKawuki, J., Chen, S., Fang, Y., Liang, X., Chan, P. S.-F., & Wang, Z. J. (2023) · Vaccines · doi:10.3390/vaccines11050886 · Cited by 16
“The quality of the included articles was assessed using the Mixed-Method Appraisal Tool (MMAT) version 2018, which has a detailed description of the rating [49].”
MethodRichards, C., Holmes, M. E., Nash, R., & Ward, A. (2023) · Teaching and learning in nursing · doi:10.1016/j.teln.2023.03.017 · Cited by 19
A systematic review of gamification in organizational learning
Khodabandelou, R., Roghanian, P., Gheysari, H., & Amoozegar, A. (2022) · The Learning Organization · doi:10.1108/tlo-05-2022-0057 · Cited by 35
The immaturity of patient engagement in value-based healthcare—A systematic review
van der Voorden, M., Sipma, W. S., de Jong, M. F. C., Franx, A., & Ahaus, K. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1144027 · Cited by 27
“MMAT facilitates the appraisal of five research categories: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies.”
BackgroundMethodInfluentialCenso, G. D., Delfabbro, P., & King, D. L. (2023) · International Gambling Studies · doi:10.1080/14459795.2023.2199050 · Cited by 25
“Existing tools do not allow for direct comparisons (e.g. MMAT (Hong et al., 2018)) or are irrelevant to the current analysis’ objectives (e.g. QATSDD (Sirriyeh et al., 2012), QuADS (Harrison et al., 2021)).”
Use of extended reality in sleep health, medicine, and research: a scoping review
Goldsworthy, A., Chawla, J., Birt, J., Baumann, O., & Gough, S. (2023) · SLEEP · doi:10.1093/sleep/zsad201 · Cited by 19
“Seven out of the 31 included articles fulfilled all relevant criteria of the MMAT [32] (Table 1).”
BackgroundMethodInfluentialTechnologies for strengthening immunization coverage in India: a systematic review
Dudeja, S., Khan, T., Varughese, D. T., Abraham, S. G., Ninan, M. M., Prasad, C. L., Sarkar, R., & Kang, G. (2023) · The Lancet Regional Health - Southeast Asia · doi:10.1016/j.lansea.2023.100251 · Cited by 28
Seaw, K. M., Henry, C. J., & Bi, X. (2023) · Livers · doi:10.3390/livers3030033 · Cited by 21
Management of Discharge Instructions for Children With Medical Complexity: A Systematic Review
Glick, A. F., Farkas, J. S., Magro, J., Shah, A. V., Taye, M., Zavodovsky, V., Rodriguez, R. H., Modi, A. C., Dreyer, B. P., Famiglietti, H. S., et al. (2023) · PEDIATRICS · doi:10.1542/peds.2023-061572 · Cited by 24
Husebø, A. M. L., Søreide, J. A., Kørner, H., Storm, M., Wathne, H. B., Richardson, A. M., Morken, I. M., Urstad, K. H., Nordfonn, O. K., & Karlsen, B. (2023) · Supportive Care in Cancer · doi:10.1007/s00520-023-08191-7 · Cited by 30
“The MMAT score was not used for exclusion decisions [24].”
MethodMhango, W., Crowter, L., Michelson, D., & Gaysina, D. (2023) · BJPsych Open · doi:10.1192/bjo.2023.614 · Cited by 26
“36 Any discrepancies were resolved through discussion.”
News media framing of food poverty and insecurity in high-income countries: a rapid review
Kerins, C., Furey, S., Kerrigan, P., McCartan, A., Kelly, C. N. M., & Vaughan, E. (2023) · Health Promotion International · doi:10.1093/heapro/daad188 · Cited by 20
“The methodological quality of all included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) version 2018 for quantitative, qualitative and mixed methods research designs (Hong et al ., 2018).”
MethodSchipper-Eindhoven, S. M., de Knegt, N. C., Mevissen, L., van Loon, J., de Vries, R., Zhuniq, M., & Bekker, M. H. J. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1328310 · Cited by 21
Randell, R., McVey, L., Wright, J. M., Zaman, H., Cheong, V., Woodcock, D. M., Healey, F., Dowding, D., Gardner, P. H., Hardiker, N. R., et al. (2024) · Health and Social Care Delivery Research · doi:10.3310/jwqc5771 · Cited by 30
Riobóo-Lois, B., Padín, P. F., Rodríguez, R. G., & Diego, C. V. (2024) · Disability and health journal · doi:10.1016/j.dhjo.2024.101630 · Cited by 33
Chepkorir, J., Guillaume, D., Lee, J., Duroseau, B., Xia, Z., Wyche, S. P., Anderson, J., & Han, H. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21070872 · Cited by 26
“Each study was assessed for quality by two independent reviewers using the Mixed Methods Appraisal Tool, which is intended to assess the design, conduct, and analysis of qualitative, quantitative descriptive, and mixed methods studies quality [18].”
MethodThai, H., Audet, É. C., Koestner, R., Lepage, M., & O’Driscoll, G. A. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102471 · Cited by 25
Taylor, N., Boyland, E. J., & Hardman, C. A. (2024) · Appetite · doi:10.1016/j.appet.2024.107699 · Cited by 29
“Due to the inclusion of qualitative, quantitative and mixed methods studies, and the heterogeneity of study designs within the review, the Mixed Methods Appraisal Tool (MMAT) was used to assess the quality of included studies (Hong et al., 2018).”
MethodNursing Education for Sustainable Development: A Concept Analysis
Boakye, D. S., Kwashie, A. A., Adjorlolo, S., & Korsah, K. A. (2024) · Nursing Open · doi:10.1002/nop2.70058 · Cited by 22
“…critical appraisal checklist for systematic reviews and research synthesis (JBI 2014), Survey studies were appraised with JBI critical appraisal tool for cross-sectional studies (Moola et al. 2017) and mixed-method studies were appraised with the mixed-method appraisal tool (Hong et al. 2018).”
MethodMapping Research Conducted on Long-Term Care Facilities for Older People in Brazil: A Scoping Review
Wachholz, P. A., Oliveira, D., Hinsliff‐Smith, K., Devi, R., Bôas, P. J. F. V., Shepherd, V., Jacinto, A. F., Watanabe, H. A. W., Gordon, A. L., & Ricci, N. A. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18041522 · Cited by 22
“[17], the overall quality score was not calculated, and instead a more detailed presentation of the ratings of each criterion is provided.”
BackgroundAntisocial Personality Disorder in Bipolar Disorder: A Systematic Review
Carbone, E. A., de Filippis, R., Caroleo, M., Calabrò, G., Staltari, F. A., Destefano, L., Gaetano, R., Steardo, L., & Fazio, P. D. (2021) · Medicina · doi:10.3390/medicina57020183 · Cited by 24
“The methodology was evaluated using five criteria: qualitative, randomized controlled, nonrandomized, observational descriptive, mixed methods [55].”
Method
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.