The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Open accessAbstract
OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
From The Mental Health of Elite-Level Coaches: A Systematic Scoping Review (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodAchieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
BackgroundDetermination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
MethodIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
BackgroundPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
MethodSystematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Cited by
Sorted by citations
Items 201–300 of 1,001
Types, characteristics and anatomic location of physical signs in elder abuse: a systematic review
van Houten, M. E., Vloet, L. C. M., Pelgrim, T. C., Reijnders, U. J. L., & Berben, S. A. A. (2021) · European Geriatric Medicine · doi:10.1007/s41999-021-00550-z · Cited by 22
“To assess the quality of the mixed methods studies, a multimethod validated appraisal tool (MMAT version 2018) was used [19].”
MethodBoddy, J., Harris, C., O’Leary, P., Hohenhaus, M., Bond, C. E. W., Panagiotaros, C. V., & Holdsworth, L. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241249145 · Cited by 21
Wattanapisit, A., Tuangratananon, T., & Wattanapisit, S. (2020) · BMC Family Practice · doi:10.1186/s12875-020-01304-9 · Cited by 20
McGuier, E. A., Kolko, D. J., Klem, M. L., Feldman, J. A., Kinkler, G., Diabes, M. A., Weingart, L. R., & Wolk, C. B. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01747-w · Cited by 23
“After data has been extracted from all relevant studies, we will assess the methodological quality of each included study using the Mixed Methods Appraisal Tool (MMAT) [39].”
MethodInfluentialUse of the e-Delphi Method to Validate the Corporate Reputation Management Maturity Model (CR3M)
Głuszek, E. (2021) · Sustainability · doi:10.3390/su132112019 · Cited by 21
Mental health consumers' perspectives of physical health interventions: An integrative review
Tabvuma, T., Stanton, R., Browne, G., & Happell, B. (2022) · International Journal of Mental Health Nursing · doi:10.1111/inm.13000 · Cited by 22
Laging, B., Slocombe, G., Liu, P., Radford, K., & Gorelik, A. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104281 · Cited by 23
Al-Ghafri, Q., Radcliffe, P., & Gilchrist, G. (2023) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2023.109790 · Cited by 21
Psychological well-being of hospice staff: systematic review
Papworth, A., Ziegler, L. E., Beresford, B., Mukherjee, S., Fraser, L., Fisher, V., O’Neill, M., Golder, S., Bedendo, A., & Taylor, J. (2023) · BMJ Supportive & Palliative Care · doi:10.1136/spcare-2022-004012 · Cited by 24
Razak, R. A., Yusoff, S. M., Leng, C. H., & Marzaini, A. F. M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0293325 · Cited by 31
Luo, X., Xu, H., Zhang, Y., Liu, S., Xu, S., Xie, Y., Xiao, J., Hu, T., & Xiao, H. (2024) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2024.102570 · Cited by 21
Blamires, J., Annette, D., Napier, S., & Foster, M. (2023) · Children · doi:10.3390/children10061006 · Cited by 20
“Three authors independently assessed the quality of the included articles using the appropriate criteria from the MMAT [34].”
MethodInfluentialClinical decision support systems for maternity care: a systematic review and meta-analysis
Cockburn, N., Osborne, C., Withana, S., Elsmore, A., Nanjappa, R., South, M., Parry‐Smith, W., Taylor, B., Chandan, J. S., & Nirantharakumar, K. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2024.102822 · Cited by 20
Liljeberg, E., Payne, L., Josefsson, M. S., Söderström, L., & Einarsson, S. (2024) · Nutrition Research Reviews · doi:10.1017/s0954422424000192 · Cited by 25
Exploring the Current Applications and Effectiveness of ChatGPT in Nursing: An Integrative Review
Luo, Y., Miao, Y., Zhao, Y., Li, J., & Wu, Y. X. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16628 · Cited by 22
Fousseni, S., Ngangue, P., Barro, A., Sophie, R. W., Bihina, L. T., Ngoufack, M. N., Bayoulou, S., Kiki, G. M., & Salfo, O. (2025) · Vaccines · doi:10.3390/vaccines13030269 · Cited by 20
“The MMAT was selected for its applicability to reviews involving qualitative, quantitative, and mixed-methods studies [22].”
Guggisberg, M., Bottino, S., & Doran, C. M. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.699288 · Cited by 24
“, 2018), which has showed content validity and reliability (Hong et al., 2019).”
BackgroundFernandes, D. V., Martins, A. R., Canavarro, M. C., & Moreira, H. (2021) · Journal of Child and Family Studies · doi:10.1007/s10826-021-02175-z · Cited by 26
“According to Hong et al. (2018) and Hong et al. (2019), the MMAT can provide a more efficient appraisal by limiting to core criteria of heterogeneous methodologies.”
BackgroundCarvajal, B., White, H., Brooks, J., Thomson, A. M., & Cooke, A. (2022) · Midwifery · doi:10.1016/j.midw.2022.103363 · Cited by 20
Bray, L., Krogh, T. B., & Østergaard, D. (2023) · Education for Primary Care · doi:10.1080/14739879.2022.2161424 · Cited by 20
“Included studies underwent a quality assessment using the Mixed Methods Appraisal Tool (v18), which offers a quality appraisal tool for quantitative, qualitative and mixed-methods studies [7].”
MethodChen, X., Wang, Z., Zhou, J., & Li, Q. (2023) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2023.100231 · Cited by 21
Dissociative Symptoms in Complex Posttraumatic Stress Disorder: A Systematic Review
Hamer, R., Bestel, N., & Mackelprang, J. L. (2023) · Journal of Trauma & Dissociation · doi:10.1080/15299732.2023.2293785 · Cited by 20
“Risk of bias was evaluated using the McGill Mixed Methods Appraisal Tool (MMAT; Hong et al., 2019).”
MethodLi, W. W., Nannestad, J., Leow, T. Y. Q., & Heward, C. (2024) · Health Psychology Open · doi:10.1177/20551029241302969 · Cited by 25
“The full-text methodological quality assessment was performed by all three authors independently to assess the quality of the included studies, using the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2019).”
Medical student confidence in care of the dying and their family: a systematic review
Wells, G., Youssef, E., Winter, R., Wright, J. E., & Llewellyn, C. D. (2020) · BMJ Supportive & Palliative Care · doi:10.1136/bmjspcare-2019-001977 · Cited by 20
A systematic review of climate emotions and mental health in adults
Pitt, C., Norris, K., & Pecl, G. (2023) · Global Environmental Psychology · doi:10.5964/gep.11405 · Cited by 20
“Previous studies have shown this tool to be valid, reliable, and efficient (Hong et al., 2019; Hong, Fàbregues, et al., 2018; Hong, Gonzalez-Reyes, et al., 2018; Hong, Pluye, et al., 2018; Pace et al., 2012; Souto et al., 2015).”
MethodPiekema, L., ten Brug, A., Waninge, A., & van der Putten, A. A. J. (2024) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.13181 · Cited by 25
Houghtaling, B., Balis, L. E., Minaker, L. M., Kheshaifaty, K. H., Morgan, R. L., & Shanks, C. B. (2022) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2022.101760 · Cited by 19
Nudging Learning Behaviour: A Systematic Review
Lim, K. K., & Lee, C. S. (2022) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.712 · Cited by 26
“The Mixed-Methods Appraisal Tool (MMAT; Hong et al., 2019) was used to assess the quality of the studies and their nudge techniques were categorised according to the taxonomy by Münscher et al. (2016).”
MethodCrowley, T., Petinger, C., Azia, I. N., & van Wyk, B. E. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20032464 · Cited by 19
Younas, A., Durante, Á., & Fàbregues, S. (2023) · Journal of Mixed Methods Research · doi:10.1177/15586898231191441 · Cited by 25
“…(F`abregues et al., 2020; Granikov et al., 2020; Harrison et al., 2020; Younas et al., 2019a) and quality appraisal tools (F`abregues et al., 2019; Hong et al., 2019) have emphasized the importance of providing an explicit rationale for using an MMR design, formulating an MMR question and problem…”
Self-citationParent–Child Communication About Potentially Traumatic Events: A Systematic Review
Sloover, M., Stoltz, S. E. M. J., & van Ee, E. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231207906 · Cited by 26
Riobóo-Lois, B., Padín, P. F., Rodríguez, R. G., & Diego, C. V. (2024) · Disability and health journal · doi:10.1016/j.dhjo.2024.101630 · Cited by 19
Villeneuve, Y., Courtemanche, F., Firoozi, F., Gilbert, S., Desbiens, M.-P., Desjardins, A., Dinh, C., LeBlanc, V. C., & Attia, A. (2020) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2020.11.006 · Cited by 19
Mudholkar, A., Korostenski, L., Blackwell, D., & Lane, A. E. (2022) · Child Care Health and Development · doi:10.1111/cch.13005 · Cited by 22
“Although quality assessment is not a priority in scoping reviews (Pham et al., 2014), the quality of the included primary studies was appraised using the Mixed Method Appraisal Tool (MMAT)—version 2018 (Hong et al., 2019).”
BackgroundMethodKitto, S. C., Fantaye, A. W., Zevin, B., Fowler, A. C., Sachdeva, A. K., & Raîche, I. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.02.011 · Cited by 22
Rubbai, Y., Chong, M. C., Tang, L. Y., Abdullah, K. L., Mohammad, W. T., Mohajer, S., & Namazinia, M. (2024) · BMC Palliative Care · doi:10.1186/s12904-024-01543-y · Cited by 18
Aliu, A. E., Kerrison, R., & Marcu, A. (2025) · Psycho-Oncology · doi:10.1002/pon.70093 · Cited by 18
Cipriani, G. E., Molfese, S., Giovannelli, F., Güntekin, B., Vitali, N., Marcato, R., & Amanzio, M. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106121 · Cited by 21
“No overall score was calculated for the studies, in line with recommendations from Hong et al. (2018) and Glenny (2005), as these scores can “ obscure ” specific methodological strengths and weaknesses (Hong et al., 2019; Viswa-nathan et al., 2012).”
Kellerhuis, B. E., Jenniskens, K., Kusters, M. P. T., Schuit, E., Hooft, L., Moons, K. G. M., & Reitsma, J. B. (2025) · Diagnostic and Prognostic Research · doi:10.1186/s41512-025-00195-7 · Cited by 21
Shared decision making in high-grade glioma patients—a systematic review
von Essen, H. S., Piil, K., Steffensen, K. D., & Poulsen, F. R. (2020) · Neuro-Oncology Practice · doi:10.1093/nop/npaa042 · Cited by 18
Zemp, M., Friedrich, A.-S. E., Schirl, J., Dantchev, S., Voracek, M., & Tran, U. S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257874 · Cited by 22
“This is in accordance with our systematic quality assessment of primary studies using the Mixed Methods Appraisal Tool [80] (see S4 Table).”
ResultBamidele, O. O., Alexis, O., Ogunsanya, M. E., Greenley, S. L., Worsley, A. J., & Mitchell, E. D. (2022) · Supportive Care in Cancer · doi:10.1007/s00520-021-06716-6 · Cited by 22
Wise, M., Ossenberg, C., & Henderson, A. (2022) · Nurse Education Today · doi:10.1016/j.nedt.2022.105296 · Cited by 18
Magwood, O., Riddle, A. Y., Petkovic, J., Lytvyn, L., Khabsa, J., Atwere, P., Akl, E. A., Campbell, P., Welch, V. A., Smith, M. P., et al. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1237 · Cited by 18
A systematic review of lifestyle and health among patrolling police officers
Valmari, E. G., Ghazinour, M., Nygren, U., & Gilenstam, K. (2022) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2022.2083013 · Cited by 25
“It has been used in many critical appraisals globally and is considered content validated and reliable [47,48].”
BackgroundGallione, C., Barisone, M., Molon, A., Pavani, M., Torgano, C., Bassi, E., & Molin, A. D. (2022) · Acta bio-medica dell'Ateneo parmense/Acta biomedica · doi:10.23750/abm.v93i3.12942 · Cited by 31
“Senananyake et Full texts of the eligible studies were retrieved and then critically appraised for methodological quality using the Mixed Method Appraisal Tool (MMAT) (18,19).”
MethodMiao, M., Rietdijk, R., Brunner, M., Debono, D. S., Togher, L., & Power, E. (2022) · Journal of Medical Internet Research · doi:10.2196/38100 · Cited by 21
“%) studies passed all 5 questions in the MMAT, 22% (13/60) of studies passed 4 questions, and 20% (12/60) of studies passed 3 questions.”
MethodInfluentialIntegrating social support into interventions among the elderly in nursing homes: a scoping review
Behrendt, D., Spieker, S., Sumngern, C., & Wendschuh, V. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-071962 · Cited by 25
Hospice and Palliative Care during Disasters: A Systematic Review
Plagg, B., Ballmann, J., & Ewers, M. (2023) · Healthcare · doi:10.3390/healthcare11172382 · Cited by 18
Disability, equity, and measurements of livability: A scoping review
Bohorquez, N. G., Stafford, L., McPhail, S., Selim, S. M., Kularatna, S., & Malatzky, C. A. R. (2023) · Disability and health journal · doi:10.1016/j.dhjo.2023.101521 · Cited by 18
Mowbray, F., Mills, F., Symons, C., Amlôt, R., & Rubin, G. J. (2023) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.12499 · Cited by 18
Bolanis, D., Vergunst, F., Mavoa, S., Schmelefske, E., Khoury, B., Turecki, G., Orri, M., & Geoffroy, M. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.167451 · Cited by 19
Noyek, S. E., Jessa, J. S., Faulkner, V., Boerner, K. E., Dewan, T., Doyle, D., Genik, L. M., Grainger-Schatz, S., McMorris, C. A., McMurtry, C. M., et al. (2023) · Pain · doi:10.1097/j.pain.0000000000003066 · Cited by 18
Charles, W., van der Waal, M. B., Flach, J., Bisschop, A., van der Waal, R. X., Es-Sbai, H., & McLeod, C. J. (2023) · JMIR Research Protocols · doi:10.2196/50339 · Cited by 18
“This tool has been assessed and updated for reliability [34] and content validity [35].”
Li, W. W., McIntyre, R., Reid, C., & Chao, M. (2024) · Mindfulness · doi:10.1007/s12671-024-02440-4 · Cited by 20
“The methodological quality of the remaining 12 articles was assessed independently by the three authors (WL, RM, and CR/MC) using the Mixed Methods Appraisal Tool (MMAT) 2018 version (Hong et al., 2019).”
Roth, I., Tiedt, M., Brintz, C., Thompson‐Lastad, A., Ferguson, G. M., Agha, E., Holcomb, J. L., Gardiner, P. M., & Leeman, J. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00595-8 · Cited by 17
“To assess the quality of each article, two reviewers independently extracted data and assessed risk of bias using the Mixed Methods Appraisal Tool version 2018 [37].”
MethodBakhti, R., Daler, H., Ogunro, H., Hope, S., Hargreaves, D. S., & Nicholls, D. (2025) · Journal of Medical Internet Research · doi:10.2196/68544 · Cited by 17
“The quality assessment was conducted using the Mixed Methods Appraisal Tool (MMAT) from McGill University [18], due to the broad spectrum of methodologies used by the included studies.”
A Meta-Synthesis of Policy Recommendations Regarding Human Mobility in the Context of Climate Change
Schwerdtle, P. N., Stockemer, J., Bowen, K., Sauerborn, R., McMichael, C., & Danquah, I. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17249342 · Cited by 19
Singh, H., Tang, T., Gray, C. S., Kokorelias, K. M., Thombs, R., Plett, D., Heffernan, M. G., Jarach, C. M., Armas, A., Law, S., et al. (2022) · JMIR Aging · doi:10.2196/35929 · Cited by 22
Raemdonck, E., Lambotte, D. F., Witte, N. D., & Gorus, E. (2022) · Health & Social Care in the Community · doi:10.1111/hsc.13928 · Cited by 17
Adesanya, A. M., Barrett, S., Moffat, M., Aquino, M. R. J., Nicholson, W., TURNER, G. M., Cook, E., Tyndall, S., & Rankin, J. M. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-066963 · Cited by 18
Patient Preferences for Cardiac Rehabilitation – A Systematic Review
Liu, Y., Su, M., Lei, Y., Tian, J., Xue, L., & Zhang, L. (2023) · Patient Preference and Adherence · doi:10.2147/ppa.s392417 · Cited by 17
Bou, C. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20064807 · Cited by 17
Jenson, R. J., Lee, M. S., Day, A. D., Hughes, A. E., Maroushek, E. E., & Roberts, K. D. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02278-2 · Cited by 17
Wall, G., Isbel, S. T., Gustafsson, L. A., & Pearce, C. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2236021 · Cited by 17
“The MMAT has undergone multiple revisions since its design in 2006 to improve its content validity [21].”
MethodDomestic Violence Against Women in Nepal: A Systematic Review of Risk Factors
Sapkota, B. D., Simkhada, P. P., Newton, D., & Parker, S. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380231222230 · Cited by 24
Wu, M., Pu, L., Grealish, L. A., Jones, C., & Moyle, W. (2020) · Journal of Clinical Nursing · doi:10.1111/jocn.15198 · Cited by 17
“The MMAT was employed because it is a valid and reliable critical appraisal tool designed for systematic mixed methods studies (Hong et al., 2019).”
MethodMason, M., Wallis, M. C., Lord, B., & Barr, N. (2020) · Australasian Emergency Care · doi:10.1016/j.auec.2020.06.004 · Cited by 18
Carter, B., Roland, D., Bray, L., Harris, J., Pandey, P., Fox, J., Carrol, E. D., & Neill, S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236013 · Cited by 17
“The MMAT facilitates the critical appraisal process "by providing, within a single tool, methodological quality criteria for different designs" [28] p57.”
MethodZullig, L. L., Drake, C., Shahsahebi, M., Avecilla, R. A. V., Whitney, C. A., Mills, C., & Oeffinger, K. C. (2022) · Journal of Cancer Survivorship · doi:10.1007/s11764-022-01212-0 · Cited by 19
“Due to various designs and methodological approaches, we used the Mixed Methods Appraisal Tool (MMAT) to assess study quality [29].”
MethodVahedi, L., Qushua, N., Seff, I., Doering, M. M., Stoll, C., Bartels, S. A., & Stark, L. (2022) · Trauma Violence & Abuse · doi:10.1177/15248380221097439 · Cited by 19
Mixed methods and the Delphi method
Taghipoorreyneh, M. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-818630-5.11078-4 · Cited by 17
Anderberg, I., Kemps, E., & Prichard, I. (2025) · Body Image · doi:10.1016/j.bodyim.2024.101836 · Cited by 16
Sasseville, M., Supper, W., Gartner, J., Layani, G., Amil, S., Sheffield, P., Gagnon, M., Hudon, C., Lambert, S., Attisso, E. A., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/63639 · Cited by 16
“The quality of the included studies was evaluated using the Mixed Methods Appraisal Tool (MMAT), which is designed for systematic reviews that synthesize data from qualitative, quantitative, and mixed methods studies [22].”
Self-citationSterile processing in low- and middle-income countries: an integrative review
Cuncannon, A., Dosani, A., & Fast, O. (2020) · Journal of Infection Prevention · doi:10.1177/1757177420947468 · Cited by 20
Singh, H., Armas, A., Law, S., Tang, T., Gray, C. S., Cunningham, H., Thombs, R., Ellen, M., Sritharan, J., Nie, J. X., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045596 · Cited by 17
Epton, T., Ghio, D., Ballard, L. M., Allen, S. F., Kassianos, A. P., Hewitt, R. M., Swainston, K., Fynn, W. I., Rowland, V., Westbrook, J., et al. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114946 · Cited by 20
School-Based Interventions for Child and Adolescent Victims of Interpersonal Violence
Wichmann, M. L.-Y., Pawils, S., Richters, J., & Metzner, F. (2022) · Trauma Violence & Abuse · doi:10.1177/15248380221078892 · Cited by 20
Timing of treatment in osteosarcoma: challenges and perspectives – a scoping review
Kim, M. S., Bolia, I. K., Iglesias, B. C., Sharf, T., Roberts, S. I., Kang, H. P., Christ, A. B., & Menendez, L. R. (2022) · BMC Cancer · doi:10.1186/s12885-022-10061-0 · Cited by 16
“MMAT grading is reported fully in Additional file 3.”
MethodInfluentialBray, L., Antoniou, P. E., Nikolaidou, M., Fides-Valero, Á., Roberts, T. E., Ahonen, O. M., Häyrinen, S., Campos-Díaz, E., & Bamidis, P. D. (2022) · International Journal of Educational Research · doi:10.1016/j.ijer.2022.102123 · Cited by 16
Talla, P. K., Inquimbert, C., Dawson, A., Zidarov, D., Bergeron, F., & Chandad, F. (2023) · JMIR Research Protocols · doi:10.2196/44218 · Cited by 16
Seaw, K. M., Henry, C. J., & Bi, X. (2023) · Livers · doi:10.3390/livers3030033 · Cited by 16
Choon, S. E., Cruz, C., Wolf, P., Jha, R. K., Fischer, K. I., Gonçalves, D. C., Hepworth, T. J., Marshall, S. R., & Gottlieb, A. B. (2023) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.19530 · Cited by 16
“24,25 This tool provides specific criteria to assess study quality according to its design, with possible scores ranging from zero (no quality criterion satisfied) to five/100% (all quality criteria satisfied).”
Nordaunet, O. M., Gjevjon, E. R., Olsson, C., Aagaard, H., & Borglin, G. (2024) · BMC Nursing · doi:10.1186/s12912-023-01675-1 · Cited by 19
“Relevant critical appraisal tools for each individual design, for example, the Critical Appraisal Skills Programme qualitative checklist (CASP) [63] and Mixed Methods Appraisal Tool (MMAT) [64] was used.”
MethodOsuh, M. E., Oke, G. A., Lilford, R. J., Osuh, J. I., Harris, B., Owoaje, E., Lawal, F. B., Omigbodun, A. O., Adedokun, B., & Chen, Y. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309319 · Cited by 18
Songs for health education and promotion: a systematic review with recommendations
Sheffield, D., & Irons, J. Y. (2021) · Public Health · doi:10.1016/j.puhe.2021.07.020 · Cited by 16
Hartley, M., Due, C., & Dorstyn, D. (2022) · Disability and Rehabilitation · doi:10.1080/09638288.2021.1921859 · Cited by 15
Shehu, E., Kugler, C. M., Schäfer, N., Rosen, D., Schaefer, C., Kötter, T., Follmann, M., & Pieper, D. (2024) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.14173 · Cited by 15
“28 Two reviewers (Eni Shehu and Charlotte M. Kugler) independently assessed the included studies.”
Ellis, L. A., Fisher, G., Saba, M., Dammery, G., Churruca, K., Spanos, S., Smith, C. L., Forrester, B., Zurynski, Y. A., & Braithwaite, J. (2025) · Learning Health Systems · doi:10.1002/lrh2.70006 · Cited by 16
Barnett, M. E., Breen, K. E., Kennedy, J. A., Hernandez, M., Matsoukas, K., & MacGregor, M. E. (2021) · Clinical Genetics · doi:10.1111/cge.14042 · Cited by 16
“For the full-text review, reviewers assessed for eligibility, reasons for exclusion, and quality using the MMAT.”
MethodInfluentialAn integrative review of the social determinants of mental health among older caregivers
Savela, R., Schwab, U., & Välimäki, T. H. (2021) · Nursing Open · doi:10.1002/nop2.1094 · Cited by 15
Peevey, N., Flores, E. C., & Seguin, M. (2022) · Global Public Health · doi:10.1080/17441692.2022.2049343 · Cited by 16
Barratt, M., Bail, K. S., & Paterson, C. C. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16429 · Cited by 15
Ivanova, D., Bishop, F. L., Newell, D., Field, J. R., & Walsh, M. (2022) · Chiropractic & Manual Therapies · doi:10.1186/s12998-022-00442-4 · Cited by 16
“MMAT focuses on assessing the methodological quality of the studies as the most essential criteria when it comes to validity of the findings [27].”
BackgroundMethodInfluentialCommunity led housing, health and wellbeing: a comprehensive literature review
McClatchey, R., McClymont, K., Griffin, E., & Carmichael, L. (2023) · International Journal of Housing Policy · doi:10.1080/19491247.2023.2232200 · Cited by 15
“The tool is recommended for rating the methodological quality of literature, and its reliability (souto et al., 2015) and content validity (Hong et al., 2019) has been corroborated. as the search also included grey literature, the quality assessment aaCODs checklist was used to rate the quality of…”
A qualitative metasummary of detransition experiences with recommendations for psychological support
Expósito-Campos, P., Pérez, J. I., & Salaberría, K. (2024) · International Journal of Clinical and Health Psychology · doi:10.1016/j.ijchp.2024.100467 · Cited by 15
“The Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2019) was used to assess the methodological quality of the studies included in the review.”
MethodVedrenne‐Gutiérrez, F., López-Suero, C. D. C., Hoyos-Bermea, A. D., Mora-Flores, L. P., Monroy‐Fraustro, D., Orozco-Castillo, M. F., Velasco, J. F. M., & Altamirano‐Bustamante, M. M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32381 · Cited by 15
“Experts have developed and validated such checklists [27 – 29].”
MethodZeng, Y., Pan, T., & Sun, M. (2024) · Medical Education Online · doi:10.1080/10872981.2024.2376802 · Cited by 15
“The same two authors were involved in the critical appraisal process by using MMAT [23].”
MethodMental Health Stigma Reduction Interventions Among Men: A Systematic Review
Sweeney, J., O’Donnell, S., Roche, E., White, P. J., Carroll, P., & Richardson, N. (2024) · American Journal of Men s Health · doi:10.1177/15579883241299353 · Cited by 16
Antibiotic use in Brazilian hospitals in the 21st century: a systematic review
Pereira, L. B., Zanetti, M. O. B., Sponchiado, L. P., Rodrigues, J. P. V., Campos, M. S. D. A., Varallo, F. R., & Pereira, L. R. L. (2021) · Revista da Sociedade Brasileira de Medicina Tropical · doi:10.1590/0037-8682-0861-2020 · Cited by 18
Metadata
- Authors
- Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 49-59.e1
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2924322406
- Cited by
- 1,082
- References
- 50
- Citations by year
- 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
- FWCI
- 168.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
- MeSH terms
- Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
