The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,801–2,900 of 4,578
Zalewski, B., Weiss, G. A., Campoy, C., Décsi, T., Profio, E. D., Mestdagh, R., Rakhshandehroo, M., Szajewska, H., Theis, S., Vaughan, E. E., et al. (2024) · Nutrition Reviews · doi:10.1093/nutrit/nuae212 · Cited by 4
Prentice, S., Patel, D. N., & Dorstyn, D. (2025) · Education for Primary Care · doi:10.1080/14739879.2025.2469494 · Cited by 6
“We used the Mixed Methods Appraisal Tool [MMAT [42]; to assess the methodological rigour of included studies.”
InfluentialDeterminants of healthy lifestyle behaviours in colorectal cancer survivors: a systematic review
Have, J. D. V.-T., Winkels, R. M., Bloemhof, S. A. G., Zondervan, A., Krabbenborg, I., Kampman, E., & Winkens, L. H. H. (2025) · Supportive Care in Cancer · doi:10.1007/s00520-025-09315-x · Cited by 4
“It is not advised by the developers of the tool to calculate an overall score of the quality [22, 23].”
InfluentialWheeler, A., Rahiri, J., Newport, R., Hanchard, S., Brewer, K., Paynter, J., Winter-Smith, J., Selak, V., Ameratunga, S. N., Grey, C., et al. (2025) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2025.101511 · Cited by 4
“32 Texts were further categorised based on the number of adapted CONSIDER criteria that were met.”
Bokhari, S. H. R., Muzaffar, M. H., Ahmad, B., Waheed, S., Rehman, S. U., & Syed, K. (2025) · Cureus · doi:10.7759/cureus.80860 · Cited by 2
“Quality assessment was conducted using the Mixed Method Appraisal Tool (MMAT) [22].”
McEvoy, D., Wilson, C., Healy, C., Mongan, D., Sheaf, G., Sweeney, C., Higgins, A., Ryan, L., Darker, C. D., Gallant, A. J., et al. (2025) · HRB Open Research · doi:10.12688/hrbopenres.14123.1 · Cited by 2
Pritchard, E., Merdian, H. L., & Gresswell, D. M. (2025) · Journal of Sexual Aggression · doi:10.1080/13552600.2025.2484294 · Cited by 2
“All studies were assessed for quality using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Kankam, K. S., Murray, L. L., Glista, D., Savundranayagam, M. Y., Teti, S. D., & Sedzro, M. T. (2025) · Disability and Rehabilitation · doi:10.1080/09638288.2025.2493209 · Cited by 2
“Quality assessment ratings were carried out by the lead author and a research assistant for each of the included studies using the Mixed Method Appraisal Tool (MMAT) Version 2018 [55].”
Hamiduzzaman, M., McLennan, V., Nisbet, G., et al. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12333-4 · Cited by 2
“A systematic search was conducted to identify literature and in order to assess the quality and rigor of the chosen papers, the Mixed Method Appraisal Tool (MMAT) was used [19].”
Muñoz-del-Pino, I. M., Matías-García, J. A., & Saavedra, J. (2025) · Psychiatric Quarterly · doi:10.1007/s11126-025-10158-2 · Cited by 3
Gallant, A. J., Lyne, J., O’Connor, K., Sheaf, G., Anand-Vembar, S., O’Keeffe, D., Wilson, C., Kartalova‐O’Doherty, Y., Doyle, L., Cannon, M. D., et al. (2025) · HRB Open Research · doi:10.12688/hrbopenres.14032.2 · Cited by 2
Sandsten, A., Lindgren, B., Strömbäck, M., Looi, G. E., Larsson, H., & Gabrielsson, S. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-098772 · Cited by 5
“39 Because of the aim of this review, no studies were excluded due to quality; instead, we present quality assessment in online supplemental table 3 and provide further reflections under the subheading Results.”
Family caregivers’ perspectives on dementia prevention: A mixed-methods systematic review
Wu, Y., Zhi, S., Zhong, Q., Song, D., Gao, S., Fang, S., Jing, B., & Sun, J. (2025) · Geriatric Nursing · doi:10.1016/j.gerinurse.2025.05.004 · Cited by 2
Harvey, E. (2025) · Advances in Neonatal Care · doi:10.1097/anc.0000000000001258 · Cited by 2
“18 The most common issue encountered was lack of blinding, which was nearly impossible given the nature of the intervention (Table 2 systematic Reviews (AMSTAR 2) criteria, a tool that assesses the quality of systematic reviews.”
Opoku-Ansah, N. A., Gallagher, J. E., & Niven, V. (2025) · BDJ Open · doi:10.1038/s41405-025-00314-y · Cited by 2
“A Mixed Methods Appraisal Tool was used to evaluate the quality of papers and bias risk [32].”
Berre, M. L., Laberge, M., & Yousefi, F. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-095201 · Cited by 2
“37 Again, any discrepancy will be resolved through consensus or by involving a third reviewer as needed.”
InfluentialRodda, S., Merkouris, S., Greenwood, C., Thomas, A. C., Lubman, D., & Dowling, N. (2025) · Addiction Science & Clinical Practice · doi:10.1186/s13722-025-00573-y · Cited by 3
“Reporting was informed by the Mixed Methods Appraisal Tool [38] as well as the Consolidated criteria for Reporting Qualitative Research (COREQ) for interviews and focus groups [39].”
Early Childhood Administrators’ Attitudes Towards Inclusive Education: A Systematic Review
Balıkcı, Ş., Gülboy, E., & Rakap, S. (2025) · Education Sciences · doi:10.3390/educsci15060734 · Cited by 13
James‐Martin, G., Davies, C., Riley, M. D., & Poelman, A. A. M. (2025) · Health Promotion Journal of Australia · doi:10.1002/hpja.70061 · Cited by 2
Knafl, K. A., Deatrick, J. A., Swallow, V. M., Wu, Y. P., & Thompson, D. I. (2025) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsaf050 · Cited by 4
Sheriff, S., Vizheh, M., Patel, R., Spanos, S., Lamprell, K., Braithwaite, J., & Long, J. C. (2025) · European Journal of Human Genetics · doi:10.1038/s41431-025-01904-y · Cited by 2
“For quality assessment of the included studies, the Mixed Methods Appraisal Tool (MMAT) was used [25].”
Girard, D., Tuinema, M., & van Wijngaarden, E. J. (2025) · Mortality · doi:10.1080/13576275.2025.2509510 · Cited by 2
“The MMAT aims to provide a more detailed presentation of the quality of articles and to take that into account when contrasting their results.”
InfluentialJaman, A., Corrales, T., Green, R., Galvin, E., Savaglio, M., Edmanson, C., & Skouteris, H. (2025) · Children and Youth Services Review · doi:10.1016/j.childyouth.2025.108487 · Cited by 3
Virtual Health Assistants in Preventive Cancer Care Communication: Systematic Review
Raisa, A., Chen, X., Bryan, E. G., Bylund, C. L., Alpert, J. M., Lok, B. C. L., Fisher, C. L., Thomas, L., & Krieger, J. L. R. (2025) · JMIR Cancer · doi:10.2196/73616 · Cited by 3
“For quality assessment, we used the mixed methods appraisal tool [41], as it allows an appraisal of qualitative, quantitative, and mixed methods studies.”
Ruslan, N. A. S. A., Mizher, H. A. A., Tawpik, N. H. N. M., & Zaini, S. (2025) · Journal of Pharmacy Practice and Research · doi:10.1002/jppr.70028 · Cited by 3
Song, J., MacQuarrie, S., Hennessey, A., & Zheng, Y. (2025) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000596 · Cited by 4
van Boxel, M. F., Kaplan, A., Ring, D., Jaarsma, E. H., Doornberg, J. N., & Obdeijn, M. C. (2025) · Journal of Hand Surgery (European Volume) · doi:10.1177/17531934251367541 · Cited by 2
“…accuracy studies (Whiting et al., 2011); the Mixed Methods Appraisal Tool (MMAT) for quantitative studies, case reports and mixed method studies (Hong et al., 2018; Pace et al., 2012); and the Joanna Briggs Institute (JBI) checklist for cost-effectiveness studies (Gomersall et al., 2015).”
Kasse, G. E., Cosh, S., Humphries, J. H., & Islam, M. S. (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70281 · Cited by 2
Wang, Z., Ma, X., Su, C., Zhang, Y., Zou, X., Balogun, M. R., Bergman, H., Liu, X., Sourial, N., & Vedel, I. (2025) · Health Policy and Planning · doi:10.1093/heapol/czaf061 · Cited by 2
“We used the Mixed Methods Appraisal Tool (MMAT version 2018) for quality assessment (Hong et al. 2018), which was developed and validated for the critical appraisal of studies with diverse designs.”
Self-citationDowney, P. M., Collins, J. C., El‐Den, S., & O’Reilly, C. L. (2025) · International Journal of Social Psychiatry · doi:10.1177/00207640251384126 · Cited by 4
“Following MMAT guidance (Hong et al., 2018), total scores were not calculated; instead, the proportion of studies that met each criterion is presented.”
Roushan, R., Kalaiah, M. K., Shastri, U., & Kumar, K. (2025) · Noise and Health · doi:10.4103/nah.nah_130_25 · Cited by 2
“[36] The key aspects assessed by the tool include the relevance of the sampling strategy, representativeness of the sample, measurement of variables, risk of non-response bias, and appropriateness of statistical analysis.”
Screening and Assessment of Gambling in Military Populations: A Systematic Review and Gap Analysis
Rayner, C. L., Treacy, S., Dighton, G., Champion, H., & Dymond, S. (2025) · Current Addiction Reports · doi:10.1007/s40429-025-00677-0 · Cited by 2
“Studies consistently report high prevalence rates of conditions such as PTSD [39 , 90–92], substance misuse [ 31, 35, 36 , 45, 66 , 77, 87], sexual trauma and risky sexual behaviours [ 35, 37, 64], and domestic violence [ 13 ].”
Hu, J., Yao, X., Huang, Z., Liang, D., Yang, D., Liu, D., Li, J., Zhang, Y., & Ma, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3792787 · Cited by 3
Mulqueeny, D. M., Nkabini, S. M., & Mashamba-Thompson, T. P. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01347-0 · Cited by 2
Mediators and theories of change in psychotherapy with adolescents: a systematic review protocol
Taubner, S., Saliba, A., Heinonen, E. O., Protić, S., Volkert, J., Adler, A., Barkauskienė, R., Cerón, S. C., Giacomo, D. D., Ioannou, Y., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-042411 · Cited by 2
Roswinabila, C. A. (2023) · Research Review · doi:10.52845/cmro/2022/5-12-5 · Cited by 2
Yoshimura, M., & Sumi, N. (2022) · Journal of International Nursing Research · doi:10.53044/jinr.2021-0001 · Cited by 2
“Excluding studies with low methodological quality is usually discouraged (Hong et al., 2018).”
BackgroundMethodWilson, C. (2021) · British Paramedic Journal · doi:10.29045/14784726.2021.3.5.4.68 · Cited by 2
“Study quality was appraised using the Mixed Methods Appraisal Tool and data analysed using narrative synthesis.”
MethodKelly, Y., O’Rourke, N., Flynn, R., Hegarty, J. M., & O’Connor, L. (2021) · HRB Open Research · doi:10.12688/hrbopenres.13212.2 · Cited by 2
Finlay, O. J. A. (2020) · doi:10.31031/rism.2020.07.000659 · Cited by 2
“Study characteristics, including research design classification, sample characteristics and quality assessment ratings (cognizant of bias) [77] are outlined in Table 2.”
BackgroundMethodInfluentialKim, S. H., Kim, J. D., & Choi, J. (2021) · Journal of Korean Critical Care Nursing · doi:10.34250/jkccn.2021.14.2.24 · Cited by 2
Aslerin, S. L., & Tingleff, E. B. (2021) · Klinisk Sygepleje · doi:10.18261/issn.1903-2285-2021-03-03 · Cited by 2
Charmer, L., Jefford, E., & Jomeen, J. (2021) · Midwifery · doi:10.1016/j.midw.2021.103127 · Cited by 2
Bradbury, M., O’Brien, C. M., Giles, N., Fenton, S. A. M., Neilson, S. J., & Duda, J. L. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053077 · Cited by 4
Dittborn, M., Portales, B., & Brierley, J. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057867 · Cited by 1
Dzobo, M., Dzinamarira, T., Kgarosi, K., & Mashamba-Thompson, T. P. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-056140 · Cited by 2
FACILITATORS AND BARRIERS OF PATIENT INVOLVEMENT IN HOSPITAL SERVICES: INTEGRATIVE REVIEW
de Souza, A. D. Z., Hoffmeister, L. V., & de Moura, G. M. S. S. (2022) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2020-0395en · Cited by 2
Kelly, N., Kilmartin, A., Lannon, K., Lee, C., McLoughlin, R., Mulvanny, L., Mohamed, O., Treacy, M., Rossi, K., & O’Connell, J. (2022) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-022-03375-2 · Cited by 4
“Quality assessment The Mixed Methods Appraisal Tool (MMAT) was used to assess the quality of the studies included in this scoping review [45].”
MethodBozzolan, M., Bighi, E., Occhi, A., Mottaran, S., Simoni, G., Valpiani, G., Bombardi, S., & Roit, M. D. (2022) · Physiotherapy Theory and Practice · doi:10.1080/09593985.2022.2142083 · Cited by 4
Li, S., Zhang, H., Leng, Y., Lei, D., Yu, Q., Li, K., Ding, M., & Lo, W. L. A. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.959917 · Cited by 2
“The quality of all included articles will be assessed by the Mixed Methods Appraisal Tool (MMAT) (49).”
MethodTelesupervision in Psychotherapy: A Bibliometric and Systematic Review
Andreucci-Annunziata, P., Mellado, A., & Vega-Muñoz, A. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192316366 · Cited by 4
“Moreover, a quality assessment of the studies was performed (see Table 4) following the criteria proposed in the Mixed Methods Appraisal Tool (MMAT) [58].”
MethodOliveira, R. C., Souto, R. Q., dos Santos, J. L. G., Reichert, A. P. D. S., Ramalho, E. L. R., & Collet, N. (2022) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6294.3788 · Cited by 2
“Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers.”
BackgroundManejo do sobrepeso e obesidade em crianças e adolescentes por enfermeiras: estudo de métodos mistos
Oliveira, R. C., Souto, R. Q., dos Santos, J. L. G., Reichert, A. P. D. S., Ramalho, E. L. R., & Collet, N. (2022) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6294.3790 · Cited by 2
Soun, S., Hunter, K. F., & Dahlke, S. (2023) · Journal of Gerontological Nursing · doi:10.3928/00989134-20230106-04 · Cited by 5
Sobantu, N. A., Tshabalala, M. D., & Chetty, V. (2023) · JMIR Research Protocols · doi:10.2196/38884 · Cited by 2
“Moreover, MMAT will be used to assess and report bias from quantitative, qualitative, and mixed methods studies [20].”
Sulaiman, N. A. A., Hassan, S. A., Shaffril, H. A. M., & Hanapi, M. S. (2023) · Pertanika journal of social science & humanities · doi:10.47836/pjssh.31.1.01 · Cited by 11
Haseldine, C., O’Donoghue, G., Kearney, P. M., Riordan, F., Cotterill, S., & McHugh, S. (2023) · HRB Open Research · doi:10.12688/hrbopenres.13702.1 · Cited by 4
“The tool has been piloted for reliability and efficiency 45,46 and was updated in 2018 37 .”
Ferreira, K. M., Balsanelli, A. P., & dos Santos, J. L. G. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6554.3937 · Cited by 2
Tamborini, M. M. D. F., Centenaro, A. P. F. C., Souto, E. N. D. S., Andres, A. T. G., Stumm, E. M. F., & Colet, C. D. F. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6797.4040 · Cited by 14
Samsunder, N., Devnarain, N., Sivro, A., et al. (2023) · Tropical Medicine and Infectious Disease · doi:10.3390/tropicalmed8120514 · Cited by 2
“Eligible articles were evaluated using the quality appraisal tool [13] and scored to assess clarity, the sampling and data collection strategy, the sample representative of the target population, measurements, the risk of non-response, and statistical analysis to address the research question.”
MethodVarela, S. M., Wells, G., Henderson, M., Collins, D., Stenhouse, L., & Subramaniam, P. (2024) · Journal of Health Services Research & Policy · doi:10.1177/13558196241231191 · Cited by 2
“21,22 Overall, the included studies were of suf fi cient quality to review, although four studies did not include adequate information to determine risk of bias for non-response, 24 – 27 and one study did not pass the risk of bias (non-response) quality check.”
Dimensions of Menstrual Rights for Menstruators: A Systematic Review
Valipour, N. S., Farnam, F., Abbasi, M., & Damghanian, M. (2023) · International Journal of Women s Health and Reproduction Sciences · doi:10.15296/ijwhr.2023.34 · Cited by 3
“Two members of the research team (MD, NSV) independently evaluated the quality of selected papers and documents using Mixed Methods Appraisal Tool (MMAT) (14), and the PRISMA 2020 statements were used to evaluate the papers and documents (Figure 1).”
MethodDeveloping risk profiling for firefighters: Enhancing safety and performance
Zahari, H. M., Wong, M. M. R., Ahmad, N. D. F., et al. (2024) · MethodsX · doi:10.1016/j.mex.2024.102733 · Cited by 2
Sirait, H. S., Said, F. M., & Mohamad, N. A. (2024) · International Journal of Advancement in Life Sciences Research · doi:10.31632/ijalsr.2024.v07i02.002 · Cited by 2
Attitudes toward Adolescent Mental Health Screening in a Dental Setting: A Mixed-Methods Study
Ticku, S., Nath, S., Ramesh, N., & Riedy, C. A. (2024) · JDR Clinical & Translational Research · doi:10.1177/23800844241273775 · Cited by 6
“The study was guided by the Mixed Methods Appraisal Tool (Hong et al. 2018) and given an exempt determination by the Harvard Longwood Medical Area Office of Human Research and Administration Institutional Review Board (IRB 21-1239).”
Evaluating Mental Health First Aid: an Evidence Synthesis Review of Published Empirical Studies.Preprint
Atanda, O., Callaghan, P., Vangeli, E., Reavey, P., Carter, T., Brown, S. D., Durcan, G., O’Shea, N., White, S. J., & Wood, K. V. (2020) · Research Square · doi:10.21203/rs.3.rs-124268/v1 · Cited by 3
Ost, K., Duquesne, L., Duguay, C., Traverson, L., Mathevet, I., Ridde, V., & Zinszer, K. (2021) · medRxiv · doi:10.1101/2021.02.22.21252205 · Cited by 1
Healy, M., Bamidele, O. O., & Gillen, P. A. (2020) · doi:10.21203/rs.3.rs-40587/v1 · Cited by 2
Influences on User Trust in Healthcare Artificial Intelligence (HAI): A Systematic Review (Preprint)Preprint
Jermutus, E., Kneale, D., Thomas, J., & Michie, S. (2021) · doi:10.2196/preprints.35122 · Cited by 1
Morgan, T. L., Faught, E., Ross‐White, A., Fortier, M., Duggan, M., Jain, R., Lane, K. N., Lorbergs, A. L., Maclaren, K., McFadden, T., et al. (2022) · doi:10.2196/preprints.39112 · Cited by 2
Batterbury, A. (2022) · Queensland University of Technology · doi:10.5204/thesis.eprints.234109 · Cited by 3
Hanff, A.-M. R., Pauly, C., Pauly, L., Rauschenberger, A., Leist, A. K., Krüger, R., Zeegers, M. P., & McCrum, C. L. (2022) · doi:10.31219/osf.io/gacs7 · Cited by 2
Mental health services implementation in Colombia – a systematic reviewPreprint
Garavito, G. A. A., Burgess, R. A., Sanguineti, M. C. D., Peters, L. E. R., & Juan, N. V. S. (2023) · medRxiv · doi:10.1101/2023.01.17.23284625 · Cited by 2
Royston, R., Naughton, S., Hassiotis, A., et al. (2023) · medRxiv · doi:10.1101/2023.01.18.23284725 · Cited by 2
Alvarado, N., McVey, L., Wright, J. M., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2494863/v1 · Cited by 2
McLeod, J., Flowers, P., Gibbs, J., Estcourt, C. S., & MacDonald, J. (2023) · medRxiv · doi:10.1101/2023.08.25.23294615 · Cited by 2
“of, or barriers or facilitators to, digital SRHC and GHC among LGBTQI+ youth using the Mixed Methods Appraisal Tool version 2018 (Hong et al., 2018).”
MethodHealth Systems Reforms in Bangladesh: An Analysis of the Last Three Decades
Udechukwu, T., Fofana, T. O., Carnapete, L., Haider, S. S., Hanif, S., Clech, L., & Ridde, V. (2026) · Journal of Health Management · doi:10.1101/2023.10.11.23296847 · Cited by 8
Williams, S. A., Baldeh, M., Bah, A. J., Robinson, D. R., & Adeniyi, Y. E. (2024) · medRxiv · doi:10.1101/2024.01.11.24301103 · Cited by 2
Flenady, T., Calleja, P., Sahay, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4578511/v1 · Cited by 2
Violence against women and children in Yemen: A mixed-methods systematic reviewPreprint
Shreedhar, S., Chavan, S. A., Al-Zumair, M., Naccache, M., Shreedhar, P., & Maxwell, L. (2024) · medRxiv · doi:10.1101/2024.08.01.24310001 · Cited by 4
Vogt, M., Helfenberger, N., Appenzeller‐Herzog, C., Adlbrecht, L., & Hirt, J. (2025) · medRxiv · doi:10.1101/2025.09.04.25335102 · Cited by 2
Talbot, R., Harris, C., Cable, E., Whiten, B., Warren, J. D., Varley, R. A., & Volkmer, A. (2025) · medRxiv · doi:10.1101/2025.09.30.25336993 · Cited by 2
Robleto, E., & Shehadeh, L. A. (2025) · American Journal of Physiology. Heart and Circulatory Physiology · doi:10.1152/ajpheart.00374.2025 · Cited by 2
AI-Powered Simulation for Nursing Education: Mixed Methods Systematic Review
Jiang, H., Wang, Z., Shen, W., Meng, M., Yang, D., Li, X., & Hao, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/95167 · Cited by 4
“For mixed methods studies, the Mixed Methods Appraisal Tool (MMAT) [35] was used to evaluate the appropriateness of the qualitative, quantitative, and mixed methods components, as well as the overall integration of the findings.”
The Utilisation of Payment Models Across the HIV Continuum of Care: Systematic Review of Evidence
Rua, T., Brandão, D., Nicolau, V., & Escoval, A. (2021) · AIDS and Behavior · doi:10.1007/s10461-021-03329-2 · Cited by 1
“MMAT contains 19 core criteria in a quality scoring system which are grouped into five methodological categories, according to the study design: qualitative, quantitative with randomisation, quantitative without randomisation, quantitative descriptive studies and mixed methods.”
InfluentialAdult stroke survivor’s reintegration to normal living: a scoping review protocol
Ogunlana, M. O., Govender, P., Oyewole, O. O., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01851-x · Cited by 1
Aphasia associated with non-traumatic subarachnoid haemorrhage: a systematic review
Dunn, K., Finch, E., & Rumbach, A. F. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2022.2157056 · Cited by 1
“Assessment of the methodological quality of included studies was undertaken using the Mixed Methods Appraisal Tool [31].”
MethodТороп, К., Ярмола, Н. А., Dergach, M., Kvitka, N. O., Трикоз, С., & Vasylenko, B. (2023) · EDUWEB · doi:10.46502/issn.1856-7576/2023.17.01.8 · Cited by 1
Personality assessment in military – Bright and dark sides of the assessment measures: A review
Sohail, M., Naeem, F., Javaid, M. U., & Ibikunle, A. K. (2023) · Work · doi:10.3233/wor-220454 · Cited by 3
Helminth Infection among Children Living in an Urban Area in Tropical Countries: A Systematic Review
Soffian, S. S. S., Baharom, M., Amir, S. M., Bahari, N. I., Hassan, M. R., Rahim, S. S. S. A., Jeffree, M. S., Ramdzan, A. R., Atil, A., Mokti, K., et al. (2023) · Open Access Macedonian Journal of Medical Sciences · doi:10.3889/oamjms.2023.11176 · Cited by 3
Psychological risks experienced by interpreters in the domestic violence cases: a systematic review
Guo, N., Muurlink, O. T., & Doyle, S. (2023) · Frontiers in Sociology · doi:10.3389/fsoc.2023.1139431 · Cited by 6
“A data extraction form based on Preferred Reporting Items for Systematic reviews and Meta-Analysis for Protocol (PRISMA, Tawfik et al., 2019) and the quality assessment form titled the MixedMethods Appraisal Tool (MMAT;Hong et al., 2018) was also applied to assess the articles’ quality.”
MethodDeprescribing antipsychotics in adults with psychotic disorders – a literature review
Painter, J. M. (2023) · Mental Health Practice · doi:10.7748/mhp.2023.e1670 · Cited by 1
Omam, L.-A., O’Laughlin, K. N., Bortel, T. V. V., Musa, I. H., Fung, G., Tendongfor, N., Wudiri, Z. W., Hassan, M. N., Metuge, A., Pride, Y., et al. (2023) · Journal of Global Health Reports · doi:10.29392/001c.87858 · Cited by 1
Isa, N. S., & Halim, L. (2023) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v13-i11/19499 · Cited by 1
“The five quality assessment criteria that the reviewers used as a reference were modified from recommendations made by Hong et al. (2018); 1) Are strategies for developing of pro environmental behaviors among the articles' 2) Do the justifications presented in the articles for each of the…”
MethodAljumah, L. A., & Alhaddad, F. A. (2023) · Journal of International Special Needs Education · doi:10.9782/jisne-d-22-00008 · Cited by 1
Ahmalia, R., Aljaberi, M. A. A., & Said, M. S. M. (2024) · International Journal of Advancement in Life Sciences Research · doi:10.31632/ijalsr.2024.v07i01.002 · Cited by 5
Rostron, H., Wright, J. M., Gilbert, A. W., Dillon, B., Pini, S. A., Redmond, A. C., & Livermore, P. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082515 · Cited by 1
“36 The MMAT will be effective as guidance for assessing all five types of study approaches (qualitative, quantitative randomised controlled trials, quantitative non-randomised, quantitative descriptive and mixed methods).”
MethodBasabih, M., & Widhakuswara, I. (2024) · Jurnal Ilmu Kesehatan Masyarakat · doi:10.26553/jikm.2024.15.1.1-20 · Cited by 1
Established and emerging theatre pharmacy services: a scoping review
McGowan, A., Deasy, E., Coyle, M., & O’Connell, J. (2024) · International Journal of Pharmacy Practice · doi:10.1093/ijpp/riae013.049 · Cited by 1
Azman, N. A. N., Muda, S. M., Jamani, N. A., et al. (2024) · INTERNATIONAL JOURNAL OF CARE SCHOLARS · doi:10.31436/ijcs.v7i2.354 · Cited by 1
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.