The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Open accessAbstract
OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
From The Mental Health of Elite-Level Coaches: A Systematic Scoping Review (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodAchieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
BackgroundDetermination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
MethodIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
BackgroundPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
MethodSystematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Cited by
Sorted by citations
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2020) · JBI Evidence Synthesis · doi:10.11124/jbisrir-d-19-00169 · Cited by 747
“…to develop a bespoke tool for mixed methods primary studies or identify an already existing critical appraisal tool for use in JBI MMSRs. 24,25,35,36 Additionally, with regard to critical appraisal in the integrated approach, further investigation into how the appraisal results of…”
Harrison, R., Jones, B., Gardner, P. H., & Lawton, R. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06122-y · Cited by 570
“across a diverse range of study designs [99].”
Göttgens, I., & Oertelt‐Prigione, S. (2021) · JMIR mhealth and uhealth · doi:10.2196/28102 · Cited by 445
“At present, MMAT is the most comprehensive tool available for appraising multimethod studies [125].”
MethodCassar, S., Salmon, J., Timperio, A. F., Naylor, P., van Nassau, F., Ayala, A. M. C., & Koorts, H. (2019) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-019-0876-4 · Cited by 313
Screening and Testing for Hepatitis B Virus Infection: CDC Recommendations — United States, 2023
Conners, E. E., Panagiotakopoulos, L., Hofmeister, M. G., Spradling, P. R., Hagan, L. M., Harris, A. M., Rogers-Brown, J. S., Wester, C. W., Nelson, N. P., Contributors, et al. (2023) · MMWR Recommendations and Reports · doi:10.15585/mmwr.rr7201a1 · Cited by 310
“The Mixed Methods Appraisal Tool (MMAT) was selected because it is a validated tool for assessing nonrandomized analytic and descriptive studies, which comprise most of the HBV infection prevalence literature (29).”
Whitehead, L., Talevski, J., Fatehi, F., & Beauchamp, A. (2023) · Journal of Medical Internet Research · doi:10.2196/42719 · Cited by 274
“Quality appraisal was undertaken using the 2018 version of the Mixed Methods Appraisal Tool developed for use in systematic reviews of mixed methods studies [29].”
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2021) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000282 · Cited by 214
Kaboré, S. S., Ngangue, P., Soubeiga, D., Barro, A., Pilabré, A. H., Bationo, N., Pafadnam, Y., Drabo, K. M., Hien, H., & Savadogo, L. G. B. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1014375 · Cited by 223
“It allows the appraisal of five methodological quality categories: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies (13, 14, 16, 17).”
BackgroundInfluentialSocial media’s influence on adolescents′ food choices: A mixed studies systematic literature review
Kucharczuk, A. J., Oliver, T. L., & Dowdell, E. B. (2021) · Appetite · doi:10.1016/j.appet.2021.105765 · Cited by 203
Chabbert, M., Panagiotou, D., & Wendland, J. (2021) · Journal of Reproductive and Infant Psychology · doi:10.1080/02646838.2020.1748582 · Cited by 161
“The latest version of this tool was used for this review (Hong et al., 2019).”
Vo, V., Chen, G., Aquino, Y. S. J., Carter, S. M., Nga, Q., & Woode, M. E. (2023) · Social Science & Medicine · doi:10.1016/j.socscimed.2023.116357 · Cited by 180
Krzeczkowska, A., Spalding, D. M., McGeown, W. J., Gow, A. J., Carlson, M. C., & Brown, L. A. (2021) · Ageing Research Reviews · doi:10.1016/j.arr.2021.101400 · Cited by 186
Wong, E., Mavondo, F., & Fisher, J. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-05383-3 · Cited by 189
“The assessment is made against ve criteria, scored as ‘Yes,’ ‘No’ or ‘Can’t tell’, and it was developed systematically (31).”
Palliative care education and its effectiveness: a systematic review
Li, W., Chhabra, J., & Singh, S. (2021) · Public Health · doi:10.1016/j.puhe.2021.02.033 · Cited by 129
Xu, J. H., Lio, A., Dhaliwal, H., Andrei, S., Balakrishnan, S., Nagani, U., & Samadder, S. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.06.070 · Cited by 151
Wilson, S. A., Byrne, P., Rodgers, S., & Maden, M. (2022) · Innovation in Aging · doi:10.1093/geroni/igac002 · Cited by 148
Pimentel, J., Ansari, U., Omer, K., Gidado, Y., Baba, M. C., Andersson, N., & Cockcroft, A. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-2852-z · Cited by 124
Chauke, G. D., Nakwafila, O., Chibi, B., Sartorius, B. K. D., & Mashamba-Thompson, T. P. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e09716 · Cited by 129
Almohanna, A. A. S., Win, K. T., & Meedya, S. (2020) · Journal of Medical Internet Research · doi:10.2196/17361 · Cited by 122
Sutter, E. D., Zaçe, D., Boccia, S., Pietro, M. L. D., Geerts, D., Borry, P., & Huys, I. (2020) · Journal of Medical Internet Research · doi:10.2196/19129 · Cited by 112
“MMAT and CASP were used to criticize study aspects such as the aim and the methodology of the study [20,21].”
MethodKhosravipour, M., Khanlari, P., Khazaie, S., Khosravipour, H., & Khazaie, H. (2021) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101427 · Cited by 103
Yopa, D. S., Massom, D. M., Kiki, G. M., Sophie, R. W., Sylvie, F., Thiam, O., Zinaba, L., & Ngangue, P. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1252428 · Cited by 95
Niazkhani, Z., Toni, E., Cheshmekaboodi, M., Georgiou, A., & Pirnejad, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01159-1 · Cited by 104
Sum, G., Nicholas, S. O., Nai, Z. L., Ding, Y. Y., & Tan, W. S. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03024-4 · Cited by 105
Astridge, B., Li, W. W., McDERMOTT, B. M., & Longhitano, C. (2023) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2023.106055 · Cited by 117
Arowosegbe, A., & Oyelade, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20021514 · Cited by 101
Hudon, C., Chouinard, M., Pluye, P., Sherif, R. E., Bush, P. L., Rihoux, B., Poitras, M.-È., Lambert, M., Zomahoun, H. T. V., & Légaré, F. (2019) · The Annals of Family Medicine · doi:10.1370/afm.2419 · Cited by 99
Self-citationGuemghar, I., Padilha, P. P. D. O., Abdel‐Baki, A., Jutras‐Aswad, D., Paquette, J., & Pomey, M. (2022) · JMIR Mental Health · doi:10.2196/36094 · Cited by 93
Afaya, A., Konlan, K. D., & Kim, H. O. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07187-5 · Cited by 89
“The third step detailed how the articles were assessed for rigor using the Mixed Method Appraisal Tool (MMAT) version 2018 [19].”
MethodMental health and psychological wellbeing in the early stages of doctoral study: a systematic review
Jackman, P. C., Jacobs, L., Hawkins, R. M., & Sisson, K. (2022) · European Journal of Higher Education · doi:10.1080/21568235.2021.1939752 · Cited by 76
“At least one quality concern was identified with more than half of studies in the review (see online Appendix 5 for additional information) based on the MMAT criteria (Hong et al. 2019).”
BackgroundMethodInfluentialCommunication Needs of Cancer Patients and/or Caregivers: A Critical Literature Review
Li, J., Luo, X., Cao, Q., Lin, Y., Xu, Y., & Li, Q. (2020) · Journal of Oncology · doi:10.1155/2020/7432849 · Cited by 101
“MMAT is a reliable and effective evaluation tool that can simultaneously evaluate qualitative, quantitative, and mixed methods studies [27, 28].”
Interventions to reduce low-value imaging – a systematic review of interventions and outcomes
Kjelle, E., Andersen, E. R., Soril, L. J. J., van Bodegom‐Vos, L., & Hofmann, B. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07004-z · Cited by 87
“The Mixed Methods Appraisal Tool (MMAT) was used to assess the methodical quality of all included studies as it is considered to be an appropriate tool for appraisal of interventional studies of different methodologies [34].”
MethodWestwell-Roper, C. Y., To, S., Andjelic, G., Lu, C., Lin, B., Soller, L., Chan, E. S., & Stewart, S. E. (2021) · Pediatric Allergy and Immunology · doi:10.1111/pai.13695 · Cited by 102
The Impact of AI Scribes on Streamlining Clinical Documentation: A Systematic Review
Sasseville, M., Yousefi, F., Ouellet, S., Naye, F., Stefan, T., Carnovale, V., Bergeron, F., Ling, L., Gheorghiu, B., Hagens, S., et al. (2025) · Healthcare · doi:10.3390/healthcare13121447 · Cited by 86
“The quality of each included review was independently assessed by pairs of reviewers using the Mixed Methods Appraisal Tool (MMAT) [28–30], with all scores verified by a senior reviewer.”
Establishing a clinical service to prevent psychosis: What, how and when? Systematic review
de Pablo, G. S., Estradé, A., Cutroni, M., Andlauer, O., & Fusar‐Poli, P. (2021) · Translational Psychiatry · doi:10.1038/s41398-020-01165-x · Cited by 87
Brassel, S., Power, E., Campbell, A., Brunner, M., & Togher, L. (2021) · Journal of Medical Internet Research · doi:10.2196/26344 · Cited by 91
Newington, L., Wells, M., Adonis, A., Bolton, L., Saghdaoui, L. B., Coffey, M., Crow, J., Costa, O. F. D., Hughes, C., Savage, M., et al. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06354-y · Cited by 82
“Outcomes of the MMAT assessment [20] are shown in Fig.”
MethodResultInfluentialHilton, N. Z., Addison, S., Ham, E., Rodrigues, N. C., & Seto, M. C. (2021) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12781 · Cited by 87
Knitter, B., & Zemp, M. (2020) · Digital Psychology · doi:10.24989/dp.v1i1.1809 · Cited by 81
Hughes, M. C., Liu, Y., & Baumbach, A. (2021) · Gerontology and Geriatric Medicine · doi:10.1177/23337214211020164 · Cited by 80
Moulaei, K., Sharifi, H. P., Bahaadinbeigy, K., & Dinari, F. (2024) · BMC Psychiatry · doi:10.1186/s12888-024-05563-z · Cited by 85
Baldassarre, A., Lulli, L. G., Cavallo, F., Fiorini, L., Mariniello, A., Mucci, N., & Arcangeli, G. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1039680 · Cited by 78
“The quality assessment was performed using the Mixed Methods Appraisal Tool (MMAT) (44, 45) which is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative, and mixed methods studies, thus, it was suitable for our review.”
MethodMörelius, E., Robinson, S., Arabiat, D. H., & Whitehead, L. (2021) · Journal of Medical Internet Research · doi:10.2196/31665 · Cited by 79
Moyle, W., Murfield, J., & Lion, K. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104513 · Cited by 75
Digital Storytelling for Health-Related Outcomes in Older Adults: Systematic Review
Stargatt, J., Bhar, S. S., Bhowmik, J., & Mahmud, A. A. (2021) · Journal of Medical Internet Research · doi:10.2196/28113 · Cited by 76
“Notably, the studies also did not conduct follow-up assessments to explore whether the effects were sustained for greater than a week [47] following digital storytelling.”
BackgroundWalker, R., Drakeley, S. C., Welch, R., Leahy, D., & Boyle, J. A. (2020) · Sex Education · doi:10.1080/14681811.2020.1843013 · Cited by 98
Leadership Development Strategies in Interprofessional Healthcare Collaboration: A Rapid Review
Bornman, J., & Louw, B. (2023) · Journal of Healthcare Leadership · doi:10.2147/jhl.s405983 · Cited by 71
Therapeutic Use of Role-Playing Game (RPG) in Mental Health: A Scoping Review
Arenas, D. L., Viduani, A. C., & Araújo, R. B. (2022) · Simulation & Gaming · doi:10.1177/10468781211073720 · Cited by 73
“The quality of the studies was scored using a nominal scale and divided into: (1) qualitative studies; (2) randomized controlled trials (RCT); (3) non-randomized studies; (4) quantitative descriptive studies; and (5) mixedmethods studies (Hong et al., 2019).”
BackgroundMethodArtificial intelligence for diagnostics in radiology practice: a rapid systematic scoping review
Lawrence, R., Dodsworth, E., Massou, E., Sherlaw-Johnson, C., Ramsay, A. I. G., Walton, H., O'Regan, T., Gleeson, F., Crellin, N., Herbert, K., et al. (2025) · eClinicalMedicine · doi:10.1016/j.eclinm.2025.103228 · Cited by 69
McGuier, E. A., Kolko, D. J., Aarons, G. A., Schachter, A., Klem, M. L., Diabes, M. A., Weingart, L. R., Salas, E., & Wolk, C. B. (2024) · Implementation Science · doi:10.1186/s13012-024-01381-9 · Cited by 70
Hallett, N., Gayton, A., Dickenson, R., Franckel, M., & Dickens, G. L. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105845 · Cited by 73
Work environment of hospital nurses during the COVID‐19 pandemic in Brazil
dos Santos, J. L. G., Balsanelli, A. P., Freitas, E. D. O., Menegon, F. H. A., Carneiro, I. A., Lazzari, D. D., de Menezes, J. A. L., Erdmann, A. L., & Camponogara, S. (2021) · International Nursing Review · doi:10.1111/inr.12662 · Cited by 67
de Oliveira, J. L. C., de Magalhães, A. M. M., Matsuda, L. M., dos Santos, J. L. G., Souto, R. Q., Riboldi, C. D. O., & Ross, R. (2021) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2020-0603 · Cited by 80
Morisod, K., Malebranche, M. E., Marti, J., Spycher, J., Grazioli, V. S., & Bodenmann, P. (2022) · European Journal of Public Health · doi:10.1093/eurpub/ckac056 · Cited by 62
Catsaros, S. M., & Wendland, J. (2020) · Midwifery · doi:10.1016/j.midw.2020.102666 · Cited by 72
“Quality assessment was perormed using the Mixed Methods Appraisal Tool (MMAT- Verion 18, Hong et al., 2019 ).”
MethodUse of Technology to Promote Health and Wellbeing of People Who Are Homeless: A Systematic Review
Heaslip, V., Richer, S., Simkhada, B., Doğan, H., & Green, S. M. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18136845 · Cited by 59
van der Velde, M., van der Leeden, M., Geleijn, E., Veenhof, C., & Valkenet, K. (2023) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-023-01474-6 · Cited by 60
The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Sports Medicine - Open · doi:10.1186/s40798-023-00655-8 · Cited by 59
Sleep disturbances in Lewy body dementia: A systematic review
Elder, G. J., Lázár, A. S., Alfonso‐Miller, P., & Taylor, J. (2022) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5814 · Cited by 60
Inocian, E. P., Hill, M. B., Felicilda‐Reynaldo, R. F. D., Kelly, S. H., Paragas, E. D., & Turk, M. T. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103391 · Cited by 56
Factors Impacting Retention of Aged Care Workers: A Systematic Review
Thwaites, C. K., McKercher, J. P., Fetherstonhaugh, D., Blackberry, I., Gilmartin‐Thomas, J., Taylor, N. F., Bourke, S. L., Fowler‐Davis, S., Hammond, S., & Morris, M. E. (2023) · Healthcare · doi:10.3390/healthcare11233008 · Cited by 56
Sams, L., Langdown, B. L., Simons, J., & Všetečková, J. (2023) · International Journal of Sports Physical Therapy · doi:10.26603/001c.73795 · Cited by 55
Alrawashdeh, G. S., Fyffe, S., Azevedo, R. F. L., & Castillo, N. M. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100587 · Cited by 83
Fisher, V., Li, W. W., & Malabu, U. H. (2023) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12441 · Cited by 69
Calo, M., Judd, B. K., & Peiris, C. L. (2024) · Medical Education · doi:10.1111/medu.15391 · Cited by 67
“39”
Overview of the role of robots in upper limb disabilities rehabilitation: a scoping review
Moulaei, K., Bahaadinbeigy, K., Haghdoostd, A., Nezhad, M. S., & Sheikhtaheri, A. (2023) · Archives of Public Health · doi:10.1186/s13690-023-01100-8 · Cited by 51
Collins, J., & Murphy, G. H. (2021) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12954 · Cited by 50
Hasnan, S., Aggarwal, S., Mohammadi, L., & Koczwara, B. (2022) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2022.06.004 · Cited by 50
Nurses knowledge and attitude regarding pain: A systematic review and meta-analysis
Ortiz, M. I., Cuevas‐Suárez, C. E., Cariño‐Cortés, R., Navarrete-Hernández, J. D. J., & González-Montiel, C. A. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103390 · Cited by 52
Rodrigues, D. A., Fiol-deRoque, M. A., Mateos-Campos, R., Figueiras, A., Herdeiro, M. T., & Roque, F. (2024) · Digital Health · doi:10.1177/20552076231225133 · Cited by 62
“34,35 All discrepancies were resolved through discussion with a third researcher (MTH, AF or FR).”
BackgroundSmart villages concept — A bibliometric analysis and state-of-the-art literature review
Bokun, K., & Nazarko, J. (2023) · Progress in Planning · doi:10.1016/j.progress.2023.100765 · Cited by 63
Parental attitudes towards mandatory vaccination; a systematic review
Smith, L., Hodson, A., & Rubin, G. J. (2021) · Vaccine · doi:10.1016/j.vaccine.2021.06.018 · Cited by 49
Graduates’ employability skills in East Africa
Ortiz, L. G., Mancini, F., Jacobetty, P., & Maina, M. F. (2021) · Journal of Teaching and Learning for Graduate Employability · doi:10.21153/jtlge2021vol12no2art988 · Cited by 50
A Comprehensive Analysis of Factors Associated with Intimate Partner Femicide: A Systematic Review
Garcia-Vergara, E., Almeda, N., Ríos, B. M., Becerra‐Alonso, D., & Fernández‐Navarro, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19127336 · Cited by 53
Li, W. W., Heward, C., Merrick, A., Astridge, B., & Leow, T. Y. Q. (2024) · Journal of Pacific Rim Psychology · doi:10.1177/18344909241252325 · Cited by 51
“The full-text assessment using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2019) was the second step for study selection and was undertaken by three authors (WL, BA, and AM).”
The Effectiveness of e-Health Solutions for Aging With Cognitive Impairment: A Systematic Review
Dequanter, S., Gagnon, M., Ndiaye, M., Gorus, E., Fobelets, M., Giguere, A. M. C., Bourbonnais, A., & Buyl, R. (2020) · The Gerontologist · doi:10.1093/geront/gnaa065 · Cited by 49
Self-citationHomer, C. S., Turkmani, S., Wilson, A. N., Vogel, J. P., Shah, M. G., Fogstad, H., & Langlois, E. V. (2022) · BMJ Global Health · doi:10.1136/bmjgh-2021-006872 · Cited by 48
“30 Patient and public involvement Patients and the public were not directly involved in this review.”
BackgroundWolff, A. C., Dresselhuis, A., Hejazi, S., Dixon, D., Gibson, D., Howard, A. F., Liva, S., Astle, B., Kirkham, S. R., Noonan, V. K., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01725-2 · Cited by 47
Digital Technologies to Prevent Social Isolation and Loneliness in Dementia: A Systematic Review
Kaur, H., Kernaghan, D., Schoonmade, L. J., Egan, K. J., & Pot, A. M. (2022) · Journal of Alzheimer s Disease · doi:10.3233/jad-220438 · Cited by 47
Gunn, V., Kreshpaj, B., Santander, N. M., Vignola, E. F., Wegman, D. H., Hogstedt, C., Ahonen, E. Q., Bodin, T., Orellana, C., Baron, S., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042232 · Cited by 49
“The MMAT tool is not as detailed and extensive as other tools designed for qualitative and quantitative studies, such as the Critical Appraisal Skills Programme (CASP) Checklists [64], but its combination of tested usefulness, reliability, and ease of application with many heterogenous studies made it a robust choice…”
MethodFàbregues, S., Sáinz, M., Romano, M., Escalante-Barrios, E., Younas, A., & López-Pérez, B.-S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.956300 · Cited by 48
Self-citationNursing pre-licensure and graduate education for LGBTQ health: A systematic review
Sherman, A. D. F., Smith, S. K., Moore, S. E., Coleman, C. L., Hughes, T. L., Dorsen, C. G., Balthazar, M. S., Klepper, M., Mukerjee, R., & Bower, K. M. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2022.12.003 · Cited by 45
Volf, K., Kelly, L. P., Bengoechea, E. G., Casey, B., Gelius, P. C., Messing, S., Forberger, S., Lakerveld, J., den Braver, N. R., Żukowska, J., et al. (2022) · International Journal of Sport Policy and Politics · doi:10.1080/19406940.2022.2127835 · Cited by 46
Zhou, F., Zhang, H., Wang, H. Y., Liu, L. F., & Zhang, X. (2024) · Aging Clinical and Experimental Research · doi:10.1007/s40520-023-02652-z · Cited by 44
“The Mixed Methods Appraisal Tool (MMAT) is used by two independent assessors to rigorously evaluate the quality of included studies [31].”
MethodClozapine prescribing barriers in the management of treatment-resistant schizophrenia
Baig, A. I., Bazargan‐Hejazi, S., Ebrahim, G., & Rodriguez-Lara, J. (2021) · Medicine · doi:10.1097/md.0000000000027694 · Cited by 44
“Data quality and risk of bias in individual studies The first 2 authors critically assessed and graded the studies for risk of bias using the Mixed Methods Appraisal Tool (MMAT),([13]) and the Measurement Tool to Assess Systematic Reviews 2.”
MethodBreast cancer screening in sub-Saharan Africa: a systematic review and ethical appraisal
Martei, Y. M., Dauda, B., & Vanderpuye, V. D. N. K. (2022) · BMC Cancer · doi:10.1186/s12885-022-09299-5 · Cited by 43
“The mixed methods appraisal tool (MMAT) [17] was used to assessed the quality of the studies included in the analysis.”
Method“Wire Syndrome” Following Bonded Orthodontic Retainers: A Systematic Review of the Literature
Charavet, C., Vives, F., Aroca, S., & Dridi, S. (2022) · Healthcare · doi:10.3390/healthcare10020379 · Cited by 43
“Risk of Bias Analysis Given the heterogeneity of the studies selected for this SRL, a tool that can evaluate the methodological quality of different types of studies within the same SRL was employed: The “Mixed Methods Appraisal Tool” (MMAT) [9] (http://mixedmethodsappraisaltoolpublic. pbworks.com accessed on 1 Septem…”
MethodGaupholm, J., Papadopoulos, A., Asif, A., Dodd, W., & Little, M. (2022) · Nutrition and Health · doi:10.1177/02601060221112810 · Cited by 43
“The latest version of the Mixed Methods Appraisal Tool (MMAT) was used for quality assessment (Hong et al., 2019).”
MethodToward a Framework for Appraising the Quality of Integration in Mixed Methods Research
Fàbregues, S., Younas, A., Escalante‐Barrios, E. L., Molina-Azorín, J. F., & Vázquez-Miraz, P. (2024) · Journal of Mixed Methods Research · doi:10.1177/15586898241257555 · Cited by 43
“Second, the MMIQF does not limit the appraisal to a single type of quality as it combines methodological and reporting quality criteria (Hong & Pluye, 2019).”
Self-citationZamir, A., Tickle, A., & Sabin‐Farrell, R. (2022) · Journal of Clinical Psychology · doi:10.1002/jclp.23339 · Cited by 50
Murtagh, E., Calderón, A., Scanlon, D., & MacPhail, A. (2023) · European Physical Education Review · doi:10.1177/1356336x231155793 · Cited by 34
“The MMAT criteria focus on the core criteria that may hinder the validity of the findings of a study, for example, data collection methods, outcome data, and interpretation of results (Hong et al., 2019).”
BackgroundMotyer, G., Dooley, B. A., Kiely, P., & Fitzgerald, A. (2020) · Patient Education and Counseling · doi:10.1016/j.pec.2020.11.023 · Cited by 46
Collins, J., Horton, K., Ives, E. G.-S., Murphy, G. H., & Barnoux, M. F. L. (2022) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-022-05590-3 · Cited by 45
Khosravipour, M., & Khosravipour, H. (2020) · Chemosphere · doi:10.1016/j.chemosphere.2019.125680 · Cited by 41
Leow, T. Y. Q., Li, W. W., Miller, D. J., & McDERMOTT, B. M. (2024) · Journal of Mental Health · doi:10.1080/09638237.2024.2332812 · Cited by 58
“Articles that were disputed were discussed to achieve consensus about inclusion in the second step: the methodological appraisal of the full-text articles employing the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2019).”
Protocol for a systematic review of living labs in healthcare
Archibald, M. M., Wiebe, S., Rieger, K. L., Linton, J., & Woodgate, R. L. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-039246 · Cited by 39
Davies, J. F., Ikin, B., Francis, J., & McGain, F. (2023) · British Journal of Anaesthesia · doi:10.1016/j.bja.2023.05.017 · Cited by 43
Alipour, J., & Haghighi, M. H. H. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735181 · Cited by 40
Effects of Air Pollution on the Health of Older Adults during Physical Activities: Mapping Review
Andrade, A., D’Oliveira, A., de Souza, L. C., Bastos, A. C. R. D. F., Dominski, F. H., Stabile, L., & Buonanno, G. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043506 · Cited by 43
“The reliability and validity of the new version of the MMAT have been confirmed [30,31].”
Güler, A., Lee, R. C., Rojas‐Guyler, L., Lambert, J., & Smith, C. R. (2023) · Nursing Inquiry · doi:10.1111/nin.12589 · Cited by 38
Metadata
- Authors
- Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 49-59.e1
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2924322406
- Cited by
- 1,082
- References
- 50
- Citations by year
- 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
- FWCI
- 168.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
- MeSH terms
- Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
