The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Open accessAbstract
OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
From The Mental Health of Elite-Level Coaches: A Systematic Scoping Review (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodAchieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
BackgroundDetermination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
MethodIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
BackgroundPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
MethodSystematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Cited by
Sorted by citations
Items 301–400 of 1,001
To what extent are patients involved in researching safety in acute mental healthcare?
Brierley‐Jones, L., Ramsey, L., Canvin, K., Kendal, S., & Baker, J. A. (2022) · Research Involvement and Engagement · doi:10.1186/s40900-022-00337-x · Cited by 15
Electronic Discharge Communication Tools Used in Pediatric Emergency Departments: Systematic Review
Wozney, L., Curran, J., Archambault, P. M., Cassidy, C. E., Jabbour, M., Mackay, R., Newton, A. S., Plint, A. C., & Somerville, M. (2022) · JMIR Pediatrics and Parenting · doi:10.2196/36878 · Cited by 15
“Assigning studies an overall numerical score based on the ratings of each criterion is discouraged, because a single number cannot provide insight into which aspects of the study methodology are problematic [26].”
BackgroundResultInfluentialBatten, M., Kosari, S., Koerner, J., Naunton, M., & Cargo, M. (2022) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2022.05.006 · Cited by 15
Self-citationIntergenerational interventions and their impact on active aging: A systematic review
Savino, M., De Luca, L., Nocentini, A., & Menesini, E. (2026) · International Psychogeriatrics · doi:10.1016/j.inpsyc.2025.100142 · Cited by 14
Cultivating Growth: A Review of Flourishing Students in Higher Education
Faizah, F., Suminar, D. R., & Yoenanto, N. H. (2024) · Adolescents · doi:10.3390/adolescents4040041 · Cited by 23
Lin, H., Pryor, M., & Beckmann, N. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105391 · Cited by 22
Tang, X., (曾子欣), Z. Z., Huang, H., & Symonds, J. E. (2025) · ECNU Review of Education · doi:10.1177/20965311251371227 · Cited by 23
“In one systematic review of appraisal tools (over 500 tools), Hong et al. (2019) found that few of them include the comprehensive criteria for quantitative, qualitative, and mixed methods.”
Viskari, T., Koivumäki, T., Appelqvist‐Schmidlechner, K., Ståhl, T., Ruiz‐Ariza, A., & Fröjd, S. A. (2025) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.70150 · Cited by 18
Factors influencing AI acceptance in radiology: a systematic review across the radiology workflow
Verwey, J., de Zwart, B. H. C., IJzerman, M. J., Visser, J. J., & Sülz, S. (2025) · European Radiology · doi:10.1007/s00330-025-12129-4 · Cited by 16
Liu, L. Q., & Mehigan, S. (2021) · AORN Journal · doi:10.1002/aorn.13469 · Cited by 14
Shen, H.-Z., Ju, Y., & Zhu, Z. (2022) · International Journal of Environmental Research and Public Health · doi:10.2139/ssrn.4237694 · Cited by 14
Jones, S. M., West, C., Rappoport, J., & Akhtar, K. (2023) · Injury · doi:10.1016/j.injury.2023.01.034 · Cited by 14
Effects of interprofessional practice on nursing workload in hospitals: A systematic review
Cucolo, D. F., de Oliveira, J. L. C., Rossit, R. A. S., Mininel, V. A., Perroca, M. G., & da Silva, J. A. M. (2024) · The International Journal of Health Planning and Management · doi:10.1002/hpm.3779 · Cited by 21
Pereira, T. L., Rajendran, P., Nantsupawat, A., & Shorey, S. (2024) · Midwifery · doi:10.1016/j.midw.2024.103956 · Cited by 15
Guelmami, N., Aissa, M. B., Ammar, A., Dergaa, I., Trabelsi, K., & Jahrami, H. (2023) · doi:10.61838/kman.tjssm.1.1.5 · Cited by 14
Walton, H., Sherlaw‐Johnson, C., Massou, E., Ng, P. L., & Fulop, N. (2024) · Public Health · doi:10.1016/j.puhe.2024.08.002 · Cited by 14
Mapping Research Conducted on Long-Term Care Facilities for Older People in Brazil: A Scoping Review
Wachholz, P. A., Oliveira, D., Hinsliff‐Smith, K., Devi, R., Bôas, P. J. F. V., Shepherd, V., Jacinto, A. F., Watanabe, H. A. W., Gordon, A. L., & Ricci, N. A. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18041522 · Cited by 14
“The MMAT has different evaluation questions that enable it to be used to accommodate multiple study designs (qualitative, randomized controlled trials, non-randomized, quantitative descriptive, and mixed methods) [16].”
BackgroundChiba, R., Miyamoto, Y., Funakoshi, A., Hirota, M., Hayashi, Y., Yamanouchi, T., Mitsui, M., Inagaki, A., & Cheng, S. (2025) · Journal of Family Nursing · doi:10.1177/10748407241309138 · Cited by 13
“Two mixed studies had 10 or 11 “Yes” grades out of 17 items of Mixed-Methods Appraisal Tool (Hong et al., 2019), whereas an RCT study showed 6/11 “Yes” in the CASP RCT Checklist (Critical Appraisal Skills Programme, 2018).”
Birkeli, G. H., Lindahl, A. K., Hammersbøen, Å. M., Deilkås, E. C. T., & Ballangrud, R. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12680-2 · Cited by 15
Chen, Z., Ni, P., Wu, B., Ko, E., Liao, J., Lin, H., & Ma, P. (2025) · Journal of Cancer Survivorship · doi:10.1007/s11764-025-01812-6 · Cited by 13
“Two researchers (Z.Y.C. and P.Y.M.) independently assessed the risk of bias using the Mixed-Methods Appraisal Tool (MMAT) [41] for qualitative, quantitative, and mixed-methods studies.”
Hira, V., Palnati, S. R., & Bhakta, S. (2025) · Cureus · doi:10.7759/cureus.87960 · Cited by 13
Ouellet, S., Gallani, M. C. B. J., Gélinas, C., Fontaine, G., Archambault, P. M., Mercier, É., Severino, F., & Berube, M. (2022) · Nursing Open · doi:10.1002/nop2.1550 · Cited by 18
“3.3.4 | Confidence in cumulative evidence The quality of the evidence will be critically appraised by two independent reviewers (SO, FS) with the Mixed Methods Appraisal Tool (MMAT) v.2018 (Hong et al., 2018; Hong et al., 2019).”
BackgroundInfluentialImpact of natural disaster on oral health: A scoping review
Matsuda, S., Yoshimura, H., & Kawachi, I. I. (2023) · Medicine · doi:10.1097/md.0000000000033076 · Cited by 14
“[22,23]”
Chan, C. K. Y. (2023) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2023.2219422 · Cited by 13
“The Mixed Methods Appraisal Tool (MMAT), developed by Hong et al. and well-recognised for its reliability (Hong et al. 2019), was adopted as a quality checking tool to ensure the validity of outcomes of the selected publications by using separate criteria for different types of studies.”
MethodTeo, J. L., Bird, S. R., Wang, X., & Zheng, Z. (2023) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2023.105203 · Cited by 13
Moore, C., Lazarakis, S., Stenta, T., Alexander, M., Nguyen, R. P., Elliott, D. A., & Conyers, R. K. (2023) · Pharmacology Research & Perspectives · doi:10.1002/prp2.1150 · Cited by 13
Management of urinary incontinence in females by primary care providers: a systematic review
Charette, M., Pacheco‐Brousseau, L., Poitras, S., Ashton, R., & McLean, L. (2023) · British Journal of Urology · doi:10.1111/bju.16234 · Cited by 15
“Risk of bias assessment for included studies was carried out using the Mixed Methods Appraisal Tool (MMAT) [23,24].”
MethodAdu, Y., Hurley, J., & Ring, D. (2024) · Clinical Orthopaedics and Related Research · doi:10.1097/corr.0000000000003021 · Cited by 14
Smith, C. L., Fisher, G., Dharmayani, P. N. A., Wijekulasuriya, S., Ellis, L. A., Spanos, S., Dammery, G., Zurynski, Y. A., & Braithwaite, J. (2024) · BMC Medicine · doi:10.1186/s12916-024-03345-8 · Cited by 23
Aouad, K., de Wit, M. P. T., Elhaï, M., Benavent, D., Bertheussen, H., Zabalan, C., Primdahl, J., Studenic, P., & Gossec, L. (2024) · Annals of the Rheumatic Diseases · doi:10.1136/ard-2024-225567 · Cited by 13
McNeil, D. A., Elliott, S. A., Wong, A. T., Kromm, S. K., Bialy, L. M., Montesanti, S. R., Purificati‐Fuñe, A., Juul, S., Roach, P., Bromely, J., et al. (2024) · Birth · doi:10.1111/birt.12841 · Cited by 16
Zhang, H., & Li, F. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36485 · Cited by 19
Persuasive strategies in digital interventions to combat internet addiction: A systematic review
Theopilus, Y., Mahmud, A. A., Davis, H., & Octavia, J. R. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105725 · Cited by 18
Domains of wheelchair users' socio-emotional experiences: Design insights from a scoping review.
Rasoulivalajoozi, M., Cucuzzella, C., & Farhoudi, M. (2025) · Disability and Health Journal · doi:10.1016/j.dhjo.2025.101829 · Cited by 11
Baker, T., Mundell, N. L., Koorts, H., Pebole, M. M., Rosenbaum, S., Ganakas, E., & Teychenne, M. J. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01735-6 · Cited by 13
“The MMAT was developed to support the assessment of multiple study designs within a single tool and is considered a reliable appraisal tool for mixed methods research [60, 61].”
Behavior Change Techniques in Physical Activity Interventions for Multiple Sclerosis
Silveira, S. L., Huynh, T. L. T., Kidwell-Chandler, A. L., Bahmani, D. S., & Motl, R. W. (2021) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2021.01.071 · Cited by 19
Strategies to improve the quality of nurse triage in emergency departments: A systematic review
Ouellet, S., Gallani, M. C. B. J., Fontaine, G., Mercier, É., Lapierre, A., Severino, F., Gélinas, C., & Berube, M. (2025) · International Emergency Nursing · doi:10.1016/j.ienj.2025.101639 · Cited by 12
Ke, Y., Zhou, H., Chan, R. J., & Chan, A. (2022) · Journal of Cancer Survivorship · doi:10.1007/s11764-022-01230-y · Cited by 12
Night‐time detection and response in relation to deteriorating inpatients: A scoping review
Hotta, S., Ashida, K., & Tanaka, M. (2023) · Nursing in Critical Care · doi:10.1111/nicc.12917 · Cited by 12
Brake, T.-M., Dudek, V., Sauzet, O., & Razum, O. (2023) · Public health reviews · doi:10.3389/phrs.2023.1605602 · Cited by 12
McDonald, S. D. (2023) · Seminars in Perinatology · doi:10.1016/j.semperi.2023.151790 · Cited by 12
Conlin, M., Hamard, M., Agrinier, N., Birgand, G., Castel, P., Claudot, F., Dumartin, C., Epstein, J., & Ricci, L. (2023) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2023.12.020 · Cited by 13
Midwives’ experience of telehealth and remote care: a systematic mixed methods review
Golden, B., El-Refaay, S. M. M., McLemore, M. R., Alspaugh, A., Baltzell, K., & Franck, L. S. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082060 · Cited by 12
Application of human-centered design principles to wearable exoskeletons: a systematic review
Assare-Mokwah, E., & Arunkumar, S. (2024) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2024.2415433 · Cited by 12
A scoping review of clinical communication in cancer care in Africa
DeBoer, R. J., Wabl, C. A., Mushi, B. P., Uwamahoro, P., Athanas, R., Ndoli, D. A., Rugengamanzi, E., Ngoma, M., Tuyishime, H., Mtei, D., et al. (2025) · The Oncologist · doi:10.1093/oncolo/oyaf039 · Cited by 14
Health benefits of viewing nature through windows: A meta-analysis
Soga, M., & Gaston, K. J. (2025) · BioScience · doi:10.1093/biosci/biaf089 · Cited by 17
“We assessed the quality of study design using the Mixed Methods Appraisal Tool (MMAT), which is designed to evaluate studies employing diverse methodological approaches (Hong et al. 2019 ; see supplemental appendix S3).”
Rose, L. T., & Soundy, A. (2020) · Behavioral Sciences · doi:10.3390/bs10110171 · Cited by 16
Promoting advance care planning via mediated health resources: A systematic mixed studies review
Diegelmann, S., Bidmon, S., & Terlutter, R. (2021) · Patient Education and Counseling · doi:10.1016/j.pec.2021.06.008 · Cited by 13
A Systematic Review and Lived-Experience Panel Analysis of Hopefulness in Youth Depression Treatment
Berry, C., Hodgekins, J., Michelson, D., Chapman, L. A., Chelidoni, O., Crowter, L., Sacadura, C., & Fowler, D. G. (2021) · Adolescent Research Review · doi:10.1007/s40894-021-00167-0 · Cited by 13
Dorfman, T. L., Archibald, M. M., Haykowsky, M. J., & Scott, S. D. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02249-7 · Cited by 12
Jenkins, B., Semple, T., Suss, J., & Bennell, C. (2024) · Journal of Police and Criminal Psychology · doi:10.1007/s11896-024-09647-0 · Cited by 13
“The MMAT was selected because previous research has supported its validity (e.g. Hong et al. 2019) and the inter-rater reliability of previous versions of the MMAT was fair to excellent (Pace et al. 2012; Souto et al. 2015).”
Zhang, H., Jiang, Q., Huang, G.-Y., Kako, J., Kajiwara, K., Lyu, J., & Miyashita, M. (2020) · Journal of Cancer Education · doi:10.1007/s13187-020-01890-3 · Cited by 12
Gaps in knowledge about the vaccine coverage of immunocompromised children: a scoping review
MacDonald, S. E., Palichuk, A., Slater, L., Tripp, H., Reifferscheid, L., & Burton, C. (2021) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2021.1935169 · Cited by 11
Ma, T., Kim, J., Godinho, M. A., de Leeuw, E., Clapham, K. F., Kobel, C., & Ivers, R. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192013014 · Cited by 11
Salvage treatments for locally recurrent nasopharyngeal cancer: Systematic review and meta‐analysis
Wang, D., Liao, M., Wu, J., Luo, W., Qi, S., Liu, B., & Li, J. (2022) · Head & Neck · doi:10.1002/hed.27253 · Cited by 11
“Although the 2018 version of the MMAT does not include a scoring system, it provides a detailed presentation of the criterion for each of the ratings to inform the quality of the included studies.(22) After responding to two screening questions, each included study was rated in the appropriate category of criteria as…”
MethodKeaver, L., O’Callaghan, N., LaVertu, A. E., Semple, C. J., Hughes, C., Hanna, J. R., & Ryan, L. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-023-01359-4 · Cited by 11
“This single tool allows for a quality appraisal of studies of different designs [21, 22].”
Kinoshita, K., Sato, S., & Sugimoto, D. (2023) · International Journal of Sport and Exercise Psychology · doi:10.1080/1612197x.2023.2246055 · Cited by 13
“Since the MMAT guideline suggested that the scoring of each study is discouraged by the inventors of the guideline, the current review cannot provide scores on each selected study.”
InfluentialNorris, S., Graham, L., Wilkinson, L., Savory, S., & Robinson, L. (2023) · Trauma · doi:10.1177/14604086231211995 · Cited by 13
Goulet, J., Maltais, C., Archambault, I., Noël, V., & Guériton, M. (2023) · Children and Youth Services Review · doi:10.1016/j.childyouth.2023.107388 · Cited by 27
Interventions to Improve Health Among Refugees in the United States: A Systematic Review
Bitterfeld, L., Ozkaynak, M., Denton, A. H., Normeshie, C. A., Valdez, R. S., Sharif, N., Caldwell, P. A., & Hauck, F. R. (2024) · Journal of Community Health · doi:10.1007/s10900-024-01400-2 · Cited by 11
Terceiro, L., Mustafa, M. I., Hägglund, M., & Kharko, A. (2024) · JMIR Research Protocols · doi:10.2196/65099 · Cited by 12
“Systematic reviews are considered one of the most informative sources of research evidence and have supported decision-making in health care in recent decades [40,48].”
Ancona, C., Caroppo, E., & Lellis, P. D. (2025) · Health and Technology · doi:10.1007/s12553-025-00942-2 · Cited by 19
“According to MMAT, for each study the correct category must be selected for appraisal, followed by an evaluation based on the 5 criteria specific to that category (for the specific biases investigated for each category, see [17]).”
Kukreja, B. J., & Kukreja, P. (2025) · Cureus · doi:10.7759/cureus.79350 · Cited by 11
“The critical narrative technique was used which involves synthesis using text, tables, and figures, summarizing and validating the outcomes of the evidence [18,19].”
Lo, C. K., Hew, K. F., Xu, S., Song, Y., Chen, G., & Jong, M. S. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70005 · Cited by 13
Arsenault, V., Larouche, J., Désilets, M., Hudon, M.-A., & Hudon, A. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14113785 · Cited by 11
“A validated instrument created especially for evaluating complicated and mixed-methods reviews with a variety of qualitative, quantitative, and mixed designs is the MMAT (2018) [18].”
Mah, B. L., Cibralic, S., Hanna, J., Hart, M., Loughland, C. M., & Cosh, S. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23612 · Cited by 11
Transgender people in the nursing discourse: An integrative review
Padilha, M. I., Caravaca‐Morera, J. A., Gentil, A. G. B., Petry, S., Bellaguarda, M. L. D. R., & Silva, A. (2022) · Journal of Advanced Nursing · doi:10.1111/jan.15318 · Cited by 11
Ortiz-Paredes, D., Amoako, A., Ekmekjian, T., Engler, K., Lebouché, B., & Klein, M. B. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.877585 · Cited by 11
Life Review Intervention: A Systematic Review of Its Feasibility Among Persons With Cancer
Ng, J. K.-Q., Subramaniam, P., Ismail, F., Ahmad, M., & Shamsudin, N. L. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2022.07.016 · Cited by 11
Flyum, I. R., Gjevjon, E. R., Eklund, A. J., Lærum‐Onsager, E., & Borglin, G. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061303 · Cited by 11
Delemere, E., & Liston, P. M. (2022) · Behavior and Social Issues · doi:10.1007/s42822-022-00103-4 · Cited by 11
Hillersdal, L., Nielsen, Z. E., Nørmark, A. T., Knoop, A. S., & Piil, K. (2022) · BMC Cancer · doi:10.1186/s12885-022-10066-9 · Cited by 11
Improving the Use of Simulation in Nursing Education: Protocol for a Realist Review
Meum, T. T., Slettebø, Å., & Fossum, M. (2020) · JMIR Research Protocols · doi:10.2196/16363 · Cited by 12
Chicoine, G., Côté, J., Pépin, J., Fontaine, G., Maheu‐Cadotte, M., Hong, Q. N., Rouleau, G., Ziegler, D., & Jutras‐Aswad, D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01832-0 · Cited by 13
“According to the MMAT recommendations [54, 58, 61], the rating and rationale for each criterion of all included studies will be reported in table form.”
MethodSelf-citationTheoretical Analysis of the Cycle of Intimate Partner Violence: A Systematic Review
McCloud, B., & Abdullah, A. (2025) · Trauma, Violence, & Abuse · doi:10.1177/15248380241301781 · Cited by 11
Álvarez-Vázquez, J., Casal-Guisande, M., Fernández-García, A., Mosteiro, M., Torres‐Durán, M., & Fernández‐Villar, A. (2025) · Healthcare · doi:10.3390/healthcare13222933 · Cited by 16
“To assess methodological quality and risk of bias, we employed the Mixed Methods Appraisal Tool (MMAT) [57,58], a validated instrument for evaluating heterogeneous research methodologies.”
Drewniok, N., Kiselev, J., Daum, N., Mörgeli, R., Spies, C., & Schaller, S. J. (2023) · Journal of Bodywork and Movement Therapies · doi:10.1016/j.jbmt.2023.05.004 · Cited by 10
Hedima, E. W., & Okoro, R. N. (2023) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2023.06.011 · Cited by 10
Nutritional interventions in treating menopause-related sleep disturbances: a systematic review
Polasek, D., Santhi, N., Alfonso‐Miller, P., Walshe, I. H., Haskell‐Ramsay, C. F., & Elder, G. J. (2023) · Nutrition Reviews · doi:10.1093/nutrit/nuad113 · Cited by 10
“main aims, was evaluated using the Mixed Methods Appraisal Tool (MMAT).(22,23) Each study was assigned a”
MethodA scoping literature review of factors influencing cancer patients’ self-perceived burden
Chen, X., Wang, Z., Zhou, J., Loke, A. Y., & Li, Q. (2023) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2023.102462 · Cited by 10
Owens, J., Aboul-Enein, B. H., Bernstein, J., Dodge, E., & Kelly, P. J. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231207902 · Cited by 11
“Two reviewers (J.O., B.A-E) assessed the papers for quality using the Mixed Methods Appraisal Tool (MMAT), 2018 version (Hong et al., 2018, 2019).”
MethodBerube, M., Verret, M., Bourque, L., Côté, C. L., Guénette, L., Richard‐Denis, A., Ouellet, S., Singer, L. N., Gauthier, L. R., Gagnon, M., et al. (2024) · Pain · doi:10.1097/j.pain.0000000000003288 · Cited by 10
“52 This tool was designed for systematic reviews that include quantitative descriptive research, qualitative studies, and mixed-methods studies.”
Self-citationKinch, M., Kroll, T., Buckle, N., & Somanadhan, S. (2024) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2024.07.026 · Cited by 11
Mixed Methods Research Centering on Minoritized Students in Higher Education: A Literature Review
Hutson, B., & He, Y. (2024) · Innovative Higher Education · doi:10.1007/s10755-024-09748-y · Cited by 21
“First developed in 2018 using the Delphi method, which involves an expert consensus process, together with an extensive literature review, Hong et al. ( 2019 ) developed the MMAT to support reviewers in appraising and interpreting findings from studies that combine qualitative and quantitative…”
InfluentialAlmuntashiri, S. A., Alsubaie, F. F., & Alotaybi, M. (2025) · Cureus · doi:10.7759/cureus.78629 · Cited by 10
“For pilot and feasibility studies, the Mixed Methods Appraisal Tool (MMAT) [21] was used to evaluate methodological rigor.”
Self-Medication Practice in Mexico
Ortiz, M. I., Flores-Ceron, K. I., & Muñoz‐Pérez, V. M. (2022) · The Senior Care Pharmacist · doi:10.4140/tcp.n.2022.266 · Cited by 10
Conducta sexual en jóvenes universitarios: Estudio de revisión
Rivera, J. C. A., & Soto, J. A. R. (2022) · psicogente · doi:10.17081/psico.25.48.5500 · Cited by 10
Jurček, A., Keogh, B., Sheaf, G., Hafford-Letchfield, T., & Higgins, A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277384 · Cited by 8
“common types of methodologies which permits the critical appraisal of mixed study reviews within a single tool and specific quality criteria for mixed methods studies [42].”
MethodSiddique, S. M., Evans, C. V., Harhay, M. O., Johnson, E. S., Aysola, J., Weissman, G. E., Mull, N. K., Flores, E. J., Schmidt, H., Tipton, K. N., et al. (2023) · Health Equity · doi:10.1089/heq.2023.0035 · Cited by 10
Miller, R., Ong, K. I. C., Choi, S., Shibanuma, A., & Jimba, M. (2020) · BMC Public Health · doi:10.1186/s12889-020-09381-2 · Cited by 10
Shin, H. D., Cassidy, C. E., Weeks, L. E., Campbell, L. A., Drake, E. K., Wong, H. W. L., Donnelly, L., Dorey, R., Kang, H., & Curran, J. (2021) · JBI Evidence Synthesis · doi:10.11124/jbies-21-00149 · Cited by 11
Leahy, D., & McNicholas, F. (2021) · Irish Journal of Psychological Medicine · doi:10.1017/ipm.2021.9 · Cited by 9
“This is a frequently used tool for literature analysis and systematic reviews where a variety of study designs need to be appraised: it involves a series of yes/no questions that contribute to a final score, with a higher score considered to indicate higher quality (Hong et al. 2018, 2019).”
MethodYameogo, A. R., Délétroz, C., Sasseville, M., Amil, S., Da, S. M. A. R., Bodenmann, P., & Gagnon, M. (2023) · JMIR Research Protocols · doi:10.2196/50798 · Cited by 10
“Quality assessment is used to describe the selected articles and to interpret the data in the synthesis [47].”
Self-citationA Comprehensive Scoping Review of Empirical Studies on Earned Secure Attachment
Filosa, M., Sharp, C., Gori, A., & Musetti, A. (2024) · Psychological Reports · doi:10.1177/00332941241277495 · Cited by 11
“All included articles were assessed for methodological quality using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2019) by two Authors.”
Expanded roles of community health workers beyond malaria in the Asia-Pacific: A systematic review
Jongdeepaisal, M., Khonputsa, P., Sirimatayanant, M., Khuenpetch, W., Harriss, E., & Maude, R. J. (2024) · PLOS Global Public Health · doi:10.1371/journal.pgph.0003113 · Cited by 11
Josefsson, M. S., Einarsson, S., Seppälä, L., Payne, L., Söderström, L., & Liljeberg, E. (2025) · Food Science & Nutrition · doi:10.1002/fsn3.4722 · Cited by 9
“A risk of bias assessment (Furuya-Kanamori et al. 2021) of the included articles was carried out by two other researchers (LSö and MSJ) using the Mixed Methods Appraisal Tool (MMAT version 2018) (Hong et al. 2019).”
How do parents decide on genetic testing in pediatrics? A systematic review
Doll, E. S., Lerch, S. P., Schmalenberger, K. M., Alex, K., Kölker, S., Brennenstuhl, H., Pereira, S., Smith, H. S., Winkler, E. C., Mahal, J., et al. (2025) · Genetics in Medicine · doi:10.1016/j.gim.2025.101390 · Cited by 9
Salman, M., Cotton, A., Humphrey, A., Domun, T., Cheng, C., & Judah, G. (2025) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2024-003127 · Cited by 11
Gaps in Prehospital Care for Patients Exposed to a Chemical Attack – A Systematic Review
Bourassa, S., Raynard, E. P., Noebert, D., Dauphin, M., Akinola, P. S., Marseilles, J., Jouvet, P. A., & Leclerc, J. (2022) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x22000401 · Cited by 9
Oral health in prison: an integrative review
Evensen, K. B., & Bull, V. H. (2022) · International Journal of Prisoner Health · doi:10.1108/ijph-08-2021-0081 · Cited by 9
Metadata
- Authors
- Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 49-59.e1
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2924322406
- Cited by
- 1,082
- References
- 50
- Citations by year
- 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
- FWCI
- 168.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
- MeSH terms
- Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
