The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Open accessAbstract
OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
From The Mental Health of Elite-Level Coaches: A Systematic Scoping Review (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodAchieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
BackgroundDetermination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
MethodIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
BackgroundPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
MethodSystematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Cited by
Sorted by citations
Items 401–500 of 1,001
Barnhart, M., McLeod, T. C. V., & Bay, R. C. (2023) · Journal of Athletic Training · doi:10.4085/1062-6050-0429.22 · Cited by 9
Flike, K., & Byrne, T. H. (2023) · Women s Health · doi:10.1177/17455057231189550 · Cited by 9
Gunn, V., Santander, N. M., Kreshpaj, B., Vignola, E. F., Wegman, D. H., Hogstedt, C., Bodin, T., Ahonen, E. Q., Baron, S., Muntaner, C., et al. (2025) · International Journal of Social Determinants of Health and Health Services · doi:10.1177/27551938241310120 · Cited by 9
“63,64 To support the interpretation of fi ndings, we used the number of “ Yes ” responses to the screening and quality assessment questions to calculate an overall rating, judging studies with 6 – 7 “ Yes ” answers as high quality, studies with 3 – 5 “ Yes ” answers as medium quality, and studies…”
Stockwell‐Smith, G., Adeleye, A. O., Chaboyer, W. P., Cooke, M. L., Phelan, M., Todd, J., & Grealish, L. A. (2020) · Journal of Clinical Nursing · doi:10.1111/jocn.15383 · Cited by 9
Prospect of low-cost energy conservation in residential energy consumption
Tyagi, R. K., & Vishwakarma, S. (2021) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijpeds.v12.i4.pp2403-2413 · Cited by 9
“The year wise split of 19 study in [22]-[40] under review is as follows: 03 (2020) + 02 (2019) + 06 (2018) + 02 (2017) + 03 (2016) + 02 (2014) + 01 (2013).”
BackgroundTechnology aspect of Electric Vehicles Initiative's social sustainability
Tyagi, R. K., & Vishwakarma, S. (2021) · Technological Sustainability · doi:10.1108/techs-09-2021-0005 · Cited by 9
Criekinge, T. V., de Grave, P. W., Luyckx, T., & Claeys, K. (2022) · Gait & Posture · doi:10.1016/j.gaitpost.2022.03.006 · Cited by 9
Bird, K. S., Stokes, N., Rivas, C. A., & Tomlinson, M. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1263 · Cited by 9
Warmelink, J. C., Marissink, L., Kroes, L., Ranjbar, F., & Henrichs, J. (2022) · Journal of Psychosomatic Obstetrics & Gynecology · doi:10.1080/0167482x.2022.2148099 · Cited by 9
“In 2018, the MMAT was further improved on content validity [30].”
BackgroundMethodPatsani, P., Parida, J., Panda, A., Jena, S., Behera, S. S., Pradhan, A., Patra, P. K., Pati, S., Kaur, H., & Acharya, S. K. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280985 · Cited by 9
“The tool is designed to include qualitative, quantitative randomized trials, quantitative non-randomized, quantitative descriptive and mixed method studies [20].”
MethodHeffernan, E., Oving, I., Barry, T., Phung, V.-H., Siriwardena, A., & Masterson, S. (2019) · HRB Open Research · doi:10.12688/hrbopenres.12969.2 · Cited by 7
“This validated tool was designed to enable systematic review authors to evaluate the quality of studies with diverse designs and paradigms (Hong et al., 2018b; Hong et al., 2019; Souto et al., 2015).”
A rapid evidence assessment of sickle cell disease educational interventions
Oti, A. E., Heyes, K., Bruce, F., & Wilmott, D. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16370 · Cited by 8
“The MMAT is a validated tool, designed to critically appraise different study designs using uniform criteria (Hong et al., 2019).”
MethodSasseville, M., Supper, W., Gartner, J., Layani, G., Amil, S., Sheffield, P., Gagnon, M., Hudon, C., Lambert, S., Attisso, E. A., et al. (2023) · JMIR Research Protocols · doi:10.2196/48155 · Cited by 10
“The quality of the included studies will be evaluated using the mixed methods appraisal tool [36]—a tool adapted to systematic reviews synthesizing data from qualitative, quantitative, and mixed methods studies.”
MethodSelf-citationEsposito, C., Masi, F. D., Dragone, M., & Bacchini, D. (2023) · Adolescent Research Review · doi:10.1007/s40894-023-00222-y · Cited by 9
“The methodological quality of the included studies was appraised using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2019), a quality assessment tool or risk of bias tool designed to evaluate each study’s methodology against five criteria, with specific criteria differing between study…”
Broden, E. G., Bailey, V., Beke, D. M., Snaman, J. M., & Moynihan, K. M. (2023) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003357 · Cited by 8
“This report presents the mixed methods analysis of results from a single-site, cross-sectional survey of PCICU staff over 2 years (July 2019–July 2021) at a quaternary pediatric hospital following the Mixed Methods Appraisal Tool ( Supplemental Table 1 , http://links.lww.com/PCC/C428) (23).”
MethodDilworth, S., Doherty, E., Mallise, C., Licata, M., Hollis, J. L., Wynne, O., Lane, C., Wolfenden, L., Wiggers, J., & Kingsland, M. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00655-z · Cited by 9
“The Mixed Methods Appraisal Tool (MMAT) [41, 42] was used to appraise the methodological quality of included studies.”
MethodBan, K., Greenfield, S., Burrows, M., Gale, N. K., & Litchfield, I. J. (2024) · British Journal of General Practice · doi:10.3399/bjgp.2024.0228 · Cited by 8
“The Critical Appraisal Skills Programme Qualitative Studies Checklist, 24 the Quality Assessment Tool for Quantitative Studies, 25 and the Mixed Methods Appraisal Tool 26 were used for qualitative, quantitative, and mixed-method design, respectively.”
West, T., Rana, J., Awan, S., & Sagot, A. J. (2025) · JAACAP Open · doi:10.1016/j.jaacop.2025.01.003 · Cited by 10
Shared decision-making interventions in neuro-oncology practice: a systematic review
Hughes, J. G., Cozzi, F. M., Phillips, V., & Price, S. J. (2025) · Journal of Neuro-Oncology · doi:10.1007/s11060-025-05141-7 · Cited by 8
Calvo‐Schimmel, A., Qanungo, S., Newman, S., & Sterba, K. R. (2021) · Canadian Oncology Nursing Journal · doi:10.5737/23688076314412429 · Cited by 8
Exploring the life contexts of patrolling police officers in the European Union – A scoping review
Valmari, E. G., Ghazinour, M., Nygren, U., & Gilenstam, K. (2022) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2022.2041088 · Cited by 9
Birch, J., & Thompson, G. A. (2023) · Journal of Music Therapy · doi:10.1093/jmt/thac015 · Cited by 8
Pakdeesatitwara, N., Clark, I. N., & Tamplin, J. (2023) · Patient Education and Counseling · doi:10.1016/j.pec.2023.108006 · Cited by 8
Taylor, M. R., Carr, S., Baynes, L., & Kujan, O. (2024) · Journal of Dental Education · doi:10.1002/jdd.13491 · Cited by 8
Nudging Public Health Behaviors to Prevent COVID-19: A Systematic Review
Lee, C. S., Lim, K. K., & Kim, H. K. (2024) · Health Communication · doi:10.1080/10410236.2024.2317567 · Cited by 10
“MMAT was designed to facilitate the concurrent assessment of five categories of study designs: qualitative , quantitative randomized controlled trial , quantitative non-randomized , quantitative descriptive, and mixed methods studies , in a standardized approach (Hong et al., 2019).”
Khatsuria, F., McMullan, C., Aiyegbusi, O., Shaw, K. L., Iqbal, R., Kinsella, F., Wilson, K., Pyatt, L., Lewis, M., Wheldon, S. M. R., et al. (2024) · The Lancet Oncology · doi:10.1016/s1470-2045(24)00256-0 · Cited by 7
Federally Qualified Health Centers as a Model to Improve Vision Health
Bai, P., Burt, S. S., Woodward, M. A., Haber, S., Newman-Casey, P. A., Henderer, J. D., Chan, R. V. P., Chen, A., Modjtahedi, B. S., Muir, K. W., et al. (2025) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.6264 · Cited by 7
Collet, M. O., Lærkner, E., Jensen, J. F., Egerod, I., Christensen, J., Jørgensen, K., Kjærgaard, R. S., Olausson, S., Wøien, H., Lange, T., et al. (2023) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.14214 · Cited by 8
“The Critical Appraisal Skills Programme tool will be applied for studies using a qualitative design,(25) and the Mixed Methods Appraisal Tool version 2018 will be applied for studies using a mixed methods approach.(26)”
MethodSmith, D. J., Mizelle, E., Leslie, S., Li, G. X., Stone, S., Stauffer, P., Smith, A., Lewis, G., Rodden, E. L., McDermott‐Levy, R., et al. (2023) · Environmental Research Health · doi:10.1088/2752-5309/acbbe6 · Cited by 8
“The MMAT tool is commonly used to evaluate studies focusing on the acceptability, satisfaction with, or barriers to implementation of interventions in the health literature [26].”
MethodEvidence of Disaster Planning by Home Care Providers: An Integrative Literature Review
Lessinnes, S., Köhler, M., & Ewers, M. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20095658 · Cited by 8
“Of the nine studies included in the review, only three received a 100% rating using the MMAT.”
MethodInfluentialCoombes, J., Holland, A. J. A., Hunter, K., Bennett‐Brook, K., Ryder, C., Finlay, S. M., Orcher, P., Scarcella, M., Briscoe, K., Forbes, D., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191811301 · Cited by 7
“The MMAT is a comprehensive tool for reviews that include qualitative, quantitative, and mixed methods studies and meets the accepted standards for validity and reliability [35,36].”
MethodAn integrative review on the maternal health literacy among maternal and child workers
Chen, S., Yue, W. W., Han, X., Luo, J., Na, L., & Yang, M. (2022) · Journal of Nursing Management · doi:10.1111/jonm.13830 · Cited by 9
An Integrated Review of Transitional Care for Families of Pre-Term Infants
Kim, J. S., & Kim, H. R. (2024) · Healthcare · doi:10.3390/healthcare12222287 · Cited by 8
González-Devesa, D., Sánchez-Lastra, M. A., Outeda-Monteagudo, B., Diz-Gómez, J. C., & Ayán, C. (2025) · Children · doi:10.3390/children12020119 · Cited by 8
“The methodological quality of comparative and single-arm studies was assessed by means of the Mixed Methods Appraisal Tool (MMAT) [15], which allows for the critical appraisal of different types of empirical designs.”
Jammal, M., Liu, K. P. Y., George, E. S., & Kolt, G. S. (2025) · Disability and health journal · doi:10.1016/j.dhjo.2025.101912 · Cited by 7
Slejko, J. F., Lavelle, T. A., Vandigo, J., Escontrías, O. A., Schoch, S. C., & Oehrlein, E. M. (2025) · Value in Health · doi:10.1016/j.jval.2025.06.014 · Cited by 7
Theoretical models to guide undergraduate medical curriculum development: an integrative review
Yin, X., Wang, L., Li, Q., Han, J., Zhou, F., & Wang, S. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07644-3 · Cited by 7
“Quantitative, qualitative, and mixed-methods studies were assessed using the 2018 version of the Mixed Methods Assessment Tool (MMAT), which facilitates the simultaneous assessment of all forms of empirical research [16].”
Che, R., Ruan, Y., Kodate, N., Shi, Y., Liu, X., Donnelly, S., Suwa, S., Yu, W., Kong, D., & Cheung, M. (2025) · Digital Health · doi:10.1177/20552076251370058 · Cited by 7
“35 Research has demonstrated that the MMAT offers a reliable and ef fi cient approach for assessing diverse study types, particularly in reviews dealing with complex interventions and heterogeneous methodologies.”
Liu, H., Yang, X., Shao, W., & Yu, X. (2025) · Educational Psychology Review · doi:10.1007/s10648-025-10062-y · Cited by 9
“Two researchers independently evaluated the methodological quality of each study using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2019).”
Tegegn, M. G., Berlie, A. B., & Utallo, A. U. (2025) · Discover Sustainability · doi:10.1007/s43621-025-01455-8 · Cited by 7
Moral distress of intensive care unit nurses: a systematic review and meta-analysis
Qu, K., Zhu, D.-J., Yang, P.-F., Huo, H.-J., Liu, H.-J., Zhou, M., Zhu, Y., Wang, Y.-H., Chen, H., & Zhu, J.-Q. (2026) · BMC Nursing · doi:10.1186/s12912-025-04274-4 · Cited by 7
McLean, K., Edwards, S., Katiba, T. C., Bartlett, J., Herrington, M. H., Evangelou, M., Henderson, M., Nolan, A., & Skouteris, H. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100493 · Cited by 10
Sasseville, M., Ouellet, S., Rhéaume, C., Couture, V., Després, P., Paquette, J., Gentelet, K., Darmon, D., Bergeron, F., & Gagnon, M. (2023) · JMIR Research Protocols · doi:10.2196/46684 · Cited by 9
Self-citationGibbs, D., Stockings, E., Larney, S., Bromberg, D. J., Shakeshaft, A. P., & Farnbach, S. (2023) · Harm Reduction Journal · doi:10.1186/s12954-023-00832-8 · Cited by 7
Development and Validation of Student Engagement Questionnaire
Imran, I., Khan, R. A., & Aslam, K. (2023) · Proceedings of Shaikh Zayed Medical Complex Lahore · doi:10.47489/szmc.v37i4.389 · Cited by 7
“It was a mixed method study 7 , conducted at the University College of Medicine and Dentistry, University of Lahore from November 2022 till May 2023 after receiving approval from “Ethical Review Board” under Reference: ERC/07/23/01.”
MethodEmpathy in family medicine postgraduate education: A mixed studies systematic review
Ortiz-Paredes, D., Henet, P. A., Desseilles, M., & Rodríguez, C. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2328324 · Cited by 7
“DOP, PAH, and CR used the 2018 version of the Mixed Methods Appraisal Tool (MMAT) (Pace et al. 2012; Hong et al. 2018a, 2018b, 2019; Pluye and Hong 2019)”
Ayorinde, A., Ghosh, I., Shaikh, J. R., Adetunji, V. O., Brown, A., Jordan, M., Gilham, E., Todkill, D., & Ashiru-Oredope, D. A. I. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1359790 · Cited by 7
“The MMAT covers five study designs (qualitative studies, randomised controlled trials, non-randomised studies, quantitative descriptive studies, and mixed methods studies) and each has five quality criteria with three response options (‘Yes’, ‘No’ or ‘Cannot tell’).”
MethodIntimate partner violence in women with cancer: An integrative review
Ni, F., Zhou, T., Wang, L., & Cai, T. (2024) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2024.100557 · Cited by 7
Hu, W., & Chan, C. K. Y. (2024) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2024.101382 · Cited by 7
Antolí-Jover, A. M., Gázquez‐López, M., Río, P. B.-D., Pérez‐Morente, M. Á., Salvador, A. M., & Álvarez‐Serrano, M. A. (2024) · Nursing Reports · doi:10.3390/nursrep14040294 · Cited by 15
Chakra, M. A., Jessup, R. L., Beauchamp, A., Azar, D., Sharma, L., Gumanti, A., Šeman, M., & Talevski, J. (2025) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvaf007 · Cited by 7
Butler, P., Nguyen, T. L., Giummarra, M. J., O’Leary, S. P., & Sterling, M. M. (2021) · Disability and Rehabilitation · doi:10.1080/09638288.2021.1988732 · Cited by 8
Smith, A., Hewitt, J., Quinn, T. J., & Robling, M. R. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01682-w · Cited by 7
Delameillieure, A., Vandekerkhof, S., Grootven, B. V., Wuyts, W. A., & Dobbels, F. (2021) · Respiratory Research · doi:10.1186/s12931-021-01815-8 · Cited by 7
“The MMAT allows a critical appraisal of various types of qualitative, quantitative or mixed-methods research designs.”
MethodInfluentialJúnior, R. O. M., Serpa, F. S., Santos, B. L. P., de Vasconcelos, C. R., da Silva, G. F., Ruzene, D. S., & Silva, D. P. (2022) · International Journal of Environmental Science and Technology · doi:10.1007/s13762-022-04057-3 · Cited by 7
Smith, M. Y., Hogan, S., Jack, S. M., Wilson, R. T., & Oremus, M. (2022) · Journal of Public Health Policy · doi:10.1057/s41271-022-00343-z · Cited by 7
“They have defined multiple mixed methods research study designs, as well as guidelines for research, and tools for appraising the quality of published mixed methods studies [18, 19].”
MethodKrishnan, A. R., Rizal, H., Lin, R. Y. S., Tanakinjal, G. H., & Rathakrishnan, B. (2022) · Information · doi:10.3390/info13050219 · Cited by 7
Kitto, S. C., Fantaye, A. W., Davies, N. P., McFadyen, A. K., Rivera, J. A., Birch, I., Barr, H., Fletcher, S., Fournier, K., & Xyrichis, A. (2022) · Journal of Interprofessional Care · doi:10.1080/13561820.2022.2097651 · Cited by 7
Zhang, H., Wu, Y., Wang, N., Sun, X., Wang, Y., & Zhang, Y. (2022) · Journal of Nursing Management · doi:10.1111/jonm.13859 · Cited by 6
“The Mixed Methods Appraisal tool (Hong et al., 2019) was used for mixed method studies.”
MethodSchulz, J., Hamilton, C., Wonnacott, E., & Murphy, V. A. (2023) · Review of Education · doi:10.1002/rev3.3413 · Cited by 11
Social Work Research and Mixed Methods: Stronger With a Quality Framework
Foote, L. A. (2023) · Research on Social Work Practice · doi:10.1177/10497315231201157 · Cited by 7
“A number of quality frameworks have been offered in the literature (e.g., Harrison et al., 2020; Hirose & Creswell, 2022; Hong et al., 2019).”
Gedik, A., van Meerten, E., Reuvers, M. J. P., Husson, O., & van der Graaf, W. T. A. (2023) · Journal of Cancer Policy · doi:10.1016/j.jcpo.2023.100455 · Cited by 7
Walker, R., Leahy, D., Welch, R., Drakeley, S. C., & Boyle, J. A. (2021) · Sex Education · doi:10.1080/14681811.2021.1946388 · Cited by 7
Heffernan, E., Oving, I., Barry, T., Phung, V., Siriwardena, A. N., & Masterson, S. (2019) · HRB Open Research · doi:10.12688/hrbopenres.12969.1 · Cited by 7
Schulze, A., Lindemann, A., Brand, F., Geppert, J., Menning, A., Stehr, P., Reifegerste, D., & Rossmann, C. (2023) · Interactive Journal of Medical Research · doi:10.2196/45258 · Cited by 6
“To critically appraise the quality of the selected studies, we used the Mixed Methods Appraisal Tool (MMAT), which was developed for the appraisal stage of complex systematic literature reviews [44].”
MethodLecours, A., Coutu, M., & Durand, M. (2023) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-023-10122-6 · Cited by 6
Vhora, A., Davies, R. L., & Rice, K. (2023) · Psychology Learning & Teaching · doi:10.1177/14757257231196707 · Cited by 6
Can digital innovation transform rural primary health care? A systematic review protocol
Nasir, B., Leedie, F., Fox, J. L., Martin, P., Alam, K., Wallis, K. A., McGrail, M. R., & Kondalsamy‐Chennakesavan, S. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-083152 · Cited by 6
“The MMAT has shown to have high validity 18 and reliability, 19 making it a useful tool for appraising literature reviews.”
MethodSyeda, F., & Pandit, U. (2024) · Cureus · doi:10.7759/cureus.64544 · Cited by 10
Plant-Based Diets and Cardiovascular Disease in Older Adults: An Integrative Literature Review
VanCleef, T. R., & Hutchens, A. (2024) · Biological Research For Nursing · doi:10.1177/10998004241268287 · Cited by 8
“This is a valid checklist that can be used for true mixed methods studies, but also for qualitative and quantitative studies separately (Hong et al., 2019).”
Valencia-Contrera, M., Barboza, V. V., Robazzi, M. L. D. C. C., Zavala, M. O. Q., Castro-Bastidas, J. D., Ardiles-Irarrázabal, R.-A., Romero, A. N. L., Vergara, S. V., Villa-Velásquez, J., Rivera-Rojas, F., et al. (2024) · Data & Metadata · doi:10.56294/dm2024.401 · Cited by 6
Moner, N., Soubelet, A., Philippine, V., & Askenazy-Gittard, F. (2024) · European Journal of Psychotraumatology · doi:10.1080/20008066.2024.2432161 · Cited by 6
Anthonj, C., Setty, K., Höser, C., Brown, J., Ihnacik, L., & Poague, K. I. H. M. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.118526 · Cited by 6
Jinqiu, L., Tan, J.-Y. B., Wang, H., Wang, T., Li, M., Terry, D. R., Taylor, M., Huang, Y., & Sharma, S. (2025) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2025.102976 · Cited by 6
Riouch, A., & Benamar, S. (2025) · Sustainability · doi:10.3390/su17219494 · Cited by 6
Li, Y., Chen, S., Dong, X., Lu, X., Chen, X., Bai, D., Luo, W., Cao, T., Song, Z., Hou, C., et al. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.70386 · Cited by 11
Hunter, S., Pham, C., Kim, K., Trinh, L. Q., Chai, A., Deshpande, S., Johl, S. K., Morgan, P. J., & Obucina, M. (2025) · International Journal of Bank Marketing · doi:10.1108/ijbm-09-2024-0580 · Cited by 6
Negi, S., Chatterjee, D., Tripathy, S., Subramaniyan, E. U., Kumar, D. T., & Mehta, V. (2026) · International Journal of Offender Therapy and Comparative Criminology · doi:10.1177/0306624x251409814 · Cited by 6
“Besides, mixed method studies were appraised using MMAT (Mixed Method Appraisal Tool) (Hong et al., 2019).”
Costa, D. S. A., Cruz, E. B., Rodrigues, A. M., Gonçalves, D. C., Gomes, L. A., Donato, H., & Nunes, C. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045358 · Cited by 6
Mahmood, H. N., Ahmed, D. R., Neldner, S., & Neuner, F. (2024) · Current Psychology · doi:10.1007/s12144-024-05863-3 · Cited by 7
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT), designed for appraising the methodological quality of qualitative, quantitative, and mixed methods studies (Hong et al., 2019 ).”
Schröder, D., Schmachtenberg, T., Heinemann, S., Müllenmeister, C., Roder, S., El-Sayed, I., Heesen, G., Königs, G., Dopfer‐Jablonka, A., Hummers, E., et al. (2024) · Journal of Primary Care & Community Health · doi:10.1177/21501319241255592 · Cited by 7
Geleto, A. B., Vasilevski, V., & Sweet, L. P. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02628-8 · Cited by 6
Impact of Atopic Dermatitis on Adolescents and Families: A Mixed‐Method Systematic Review
Vicho‐de‐la‐Fuente, N., Martinez‐Santos, A., Rodríguez‐González, R., Flórez, Á., Sheaf, G., & Coyne, I. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16652 · Cited by 8
“To evaluate the quality of the quantitative, qualitative and mixed-methods studies, we used the Mixed Methods Appraisal Tool (MMAT) (Hong et al. 2019; Souto et al. 2015).”
InfluentialEsan, O. B., Adjei, N. K., Saberian, S., Christianson, L., Mazlan, A., Khalaf, R. K. S., Towolawi, O. Y., McHale, P., Pennington, A., Geary, R. S., et al. (2025) · BMJ Public Health · doi:10.1136/bmjph-2024-001476 · Cited by 6
“21 This tool is a standardised instrument developed by McGill University, Canada, for systematic reviews.”
Behrendt, D., Schirmer, M., Wendschuh, V., & Sumngern, C. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055692 · Cited by 7
A Systematic Review of the Effects of Second-Eye Cataract Surgery on Motor Function
Sheppard, W. E. A., McCarrick, D. J., Wilkie, R. M., Baraas, R. C., & Coats, R. O. (2022) · Frontiers in Aging · doi:10.3389/fragi.2022.866823 · Cited by 6
“The characteristics of these studies are shown in Table 1 and the quality assessment (assessed using the MMAT (Hong et al., 2019)) is shown in Table 2.”
MethodInfluentialMeasurement properties of appraisal tools for mixed methods research: A systematic review protocol
Makabe, S., Suda, T., Akagawa, Y., Abe, M., & Kakai, H. (2022) · International Journal of Science and Research Archive · doi:10.30574/ijsra.2022.7.1.0185 · Cited by 6
Buckley, L., Gibson, L., Harford, K., Cornally, N., & Curtin, M. (2022) · Journal of Child Health Care · doi:10.1177/13674935221146008 · Cited by 6
“…of cohort studies; the CASP Qualitative Checklist (CASP, 2018b) was used for qualitative studies; the Joanna Briggs Institute Critical Appraisal Tools was used for case series studies (JBI, 2017), and the Mixed Methods Appraisal Tool (MMAT) was used for mixed-method studies (Hong et al., 2019).”
MethodUnmet Care Needs in Breast Cancer Survivors: An Integrative Review
Tabari-Khomeiran, R., Ahmadi, F., Jafaraghaee, F., Noveiri, M. J. S., & Mirfarhadi, N. (2023) · Journal of Holistic Nursing and Midwifery · doi:10.32598/jhnm.33.1.2272 · Cited by 6
Komenda-Schned, S., Brunevskaya, N., Moritz, P., Landskron, S. J., Novaković, I. Z., Oberleiter, S., Wurzer, J., Lueger‐Schuster, B., Salvador‐Carulla, L., & Zeilinger, E. L. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1148702 · Cited by 6
Mzimela, J. H., & Moyo, I. (2023) · Sustainability · doi:10.3390/su15129256 · Cited by 7
The roles of pets in long-term care at home: a qualitative study
Reniers, P. W. A., Leontjevas, R., Declercq, I., Enders‐Slegers, M., Gerritsen, D. L., & Hediger, K. (2023) · BMC Geriatrics · doi:10.1186/s12877-023-04416-w · Cited by 8
“Mixed Method Appraisal Tool criteria were considered while drawing up the final version of the manuscript [16, 17].”
MethodJadidi, A., Samari, B., Nejad, A. F. A., Tapak, L., & Khodaveisi, M. (2023) · International Journal of Africa Nursing Sciences · doi:10.1016/j.ijans.2023.100643 · Cited by 6
Erwin, J., Paisi, M., Witton, R., Neill, S., Burns, L., Vassallo, I., Nelder, A., Facenfield, J., Devalia, U., & Vassallo, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182312346 · Cited by 6
“The MMAT is widely used; it has been refined since its inception and is a well-accepted tool for the assessment of quality and risk of bias which has been content validated and tested for reliability [32].”
MethodCoordination Training Model Innovation for Junior Futsal Players: Aiken’s Validity
Miftachurochmah, Y., & Sukamti, E. R. (2022) · Advances in health sciences research/Advances in Health Sciences Research · doi:10.2991/ahsr.k.220106.035 · Cited by 6
Zhi, S., Ma, D., Song, D., Gao, S., Sun, J., He, M., Zhu, X., Dong, Y., Gao, Q., & Sun, J. (2023) · International Journal of Mental Health Nursing · doi:10.1111/inm.13177 · Cited by 6
Kolahdooz, F., Jang, S. L., Deck, S., Ilkiw, D., Omoro, G., Rautio, A., Pirkola, S. P., Møller, H., Ferguson, G., Evengård, B., et al. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph22010001 · Cited by 9
Soresi, J., Bertilone, C., Banks, E., Marshall, T., Murray, K., & Preen, D. B. (2025) · Health Informatics Journal · doi:10.1177/14604582251315414 · Cited by 5
Systematic review of well-being interventions for minority healthcare workers
Bafna, T. A., Malik, M. A., Choudhury, M. C., Hu, W. T., Weston, C. M., Weeks, K. R., Connors, C. A., Burhanullah, M. H., Everly, G. S., Michtalik, H. J., et al. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1531090 · Cited by 5
“…variety of study designs included in this review, two comprehensive critical appraisal tools, the Mixed Methods Appraisal Tool (MMAT, version 2018) (15, 16) and the Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Systematic Reviews and Research Syntheses (2015) (17), were utilized.”
Stern, C., Loveday, H. P., Godfrey, C. M., Pollock, D., Hong, Q. N., Rieger, K. L., Stephenson, M., Kurian, N., Fiorini, J., & Lizarondo, L. M. (2025) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00569 · Cited by 7
Self-citation
Metadata
- Authors
- Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 49-59.e1
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2924322406
- Cited by
- 1,082
- References
- 50
- Citations by year
- 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
- FWCI
- 168.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
- MeSH terms
- Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
