Cited by 1,082Delphi Technique in ResearchMeta-analysis and systematic reviewsHealth Education and Validation

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

Open access

Hong, Quan Nha, Pluye, Pierre, Fàbregues, Sergi, Bartlett, Gillian, Boardman, Felicity, Cargo, Margaret, Dagenais, Pierre, Gagnon, Marie-Pierre, Griffiths, Frances, Nicolau, Belinda, O’Cathain, Alicia, Rousseau, Marie-Claude, Vedel, Isabelle·2019·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2019.03.008

Open this paper in CoChat — free account Open PDF

Abstract

OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2019.03.008

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
Method
Achieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
Background
Determination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
Background
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
Method
Is the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
Background
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
Method
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657

Show all 50 references

Sorted by citations

Items 501–600 of 1,001

Barriers and Facilitators of Exercise Rehabilitation in Patients With Myocardial Infarction Based on an Updated Consolidated Framework for Implementation Research: A Systematic Review
Wang, Y., Pei, H., Luo, J., Guan, M., Sun, W., Wang, H., & Xu, Q. (2025) · Reviews in Cardiovascular Medicine · doi:10.31083/rcm33508 · Cited by 6
“Two researchers used the mixed methods appraisal tool (MMAT) [15], to independently evaluate the quality of the included literature.”
Playful experience, learning perception, and motivation in a cooperative escape room activity in undergraduate nursing education: A mixed-methods study
Fagundo-Rivera, J., Garrido-Bueno, M., Romero‐Castillo, R., Badillo-Sánchez, N., & Fernández-León, P. (2025) · Learning and Instruction · doi:10.1016/j.learninstruc.2025.102212 · Cited by 8
Older adults and social support in rural Canada: A rapid mixed methods systematic review to inform social prescribing
Iverson, T., Schulze, M., Gill, K., Switkowski, Y., dos Santos, Í. K., & Ashe, M. C. (2025) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2025.106022 · Cited by 6
Barriers and Facilitators to Physical Activity Adherence in People With Type 2 Diabetes: A Mixed-Methods Systematic Review Using the COM-B Model and Theoretical Domains Framework
Guo, X., Chen, H., Xin, R., Huang, J., Long, T., & Li, M. (2025) · International Journal of Behavioral Medicine · doi:10.1007/s12529-025-10400-y · Cited by 7
“The Mixed Methods Appraisal Tool (MMAT) [17] was used to assess the quality and risk of bias for each included study.”
Sexual Victimization in Contexts of Chemsex: A Scoping Review with Harm Reduction Implications
Bernal, M. R., Riquelme, A. R., Pezzi, M., Musetti, A., Íncera-Fernández, D., & Gámez‐Guadix, M. (2026) · The Journal of Sex Research · doi:10.1080/00224499.2025.2602850 · Cited by 5
“A quality assessment was performed by two authors (MB and AR) using the Mixed Method Appraisal Tool (MMAT), which contains five categories to appraise qualitative, quantitative, and mixed studies (Hong et al., 2019).”
Outcomes of total hip arthroplasty in obese patients with and without preoperative weight loss: A systematic review and meta‐analysis
Meißner, N., Ramos‐Pascual, S., Ortwig, K. A., van Rooij, F., Schrednitzki, D., Stöve, J., & Halder, A. M. (2026) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70651 · Cited by 5
“Methodological quality of the eligible studies was independently assessed by two researchers (L.N. and M.H.) using the Mixed Methods Appraisal Tool (MMAT) [27, 28].”
Identifying indicators for measuring appropriateness of antibiotic prescribing in high-income countries: a rapid systematic review
Berry, R., Catteau, L., Caris, M. G., de Boer, M. G. J., Kherabi, Y., Magrini, E., Medioli, F., Murri, R., Ng, B. Y., Peiffer‐Smadja, N., et al. (2026) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2026.01.011 · Cited by 5
Role of Expert in Validation of Information Collection Instruments for Business Purposes
Pino, M. E. M., Ordóñez, F. R. R., Ysa, R. Á. S., Llanos, D. M. J., Cruz, M., & Calderón, B. A. C. (2023) · International Journal of Professional Business Review · doi:10.26668/businessreview/2023.v8i8.3122 · Cited by 5
Comparative Analysis of Total Laparoscopic Hysterectomy Versus Non-descent Vaginal Hysterectomy for Benign Uterine Pathologies in Women: A Systematic Review
Sonkusare, A., & Dixit, P. (2024) · Cureus · doi:10.7759/cureus.62846 · Cited by 5
Patient-Reported Outcomes after Surgical, Endoscopic, or Radiological Techniques for Nutritional Support in Esophageal Cancer Patients: A Systematic Review
Fontes, F., Fernandes, D., de Almeida, A. T. S. D., de Sá, I. M., & Dinis-Ribeiro, M. J. (2024) · Current Oncology · doi:10.3390/curroncol31100460 · Cited by 5
MAPEAMENTO DO CONHECIMENTO EM REVISÕES DE ESCOPO: UM GUIA PARA PESQUISADORES E ACADÊMICOS DE ENFERMAGEM
Silva, A., Silva, V. S. E., Padilha, M. I., Petry, S., Mendes, K. D. S., & Costa, I. C. P. (2024) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2024-0074pt · Cited by 5
Beyond Earth: Comparing Rational and Behavioral Mediators in Space Tourism Intentions
Huang, Z., Peng, K.-L., Wang, Y., & Zhang, H. Q. (2025) · Journal of Quality Assurance in Hospitality & Tourism · doi:10.1080/1528008x.2025.2504448 · Cited by 5
Mental health reported in adult invasive home mechanical ventilation through a tracheostomy: A scoping review
Pedersen, M. L., Handberg, C., & Dreyer, P. (2022) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2022.100110 · Cited by 5
Adoption of mHealth Technologies by Community Health Workers to Improve the Use of Maternal Health Services in Sub-Saharan Africa: Protocol for a Mixed Method Systematic Review
Kachimanga, C., Divala, T. H., Ket, J. C. F., Kulinkina, A. V., Zaniku, H. R., Murkherjee, J., Palazuelos, D., Abejirinde, I. O., & van den Akker, T. (2023) · JMIR Research Protocols · doi:10.2196/44066 · Cited by 7
Behavioral assessment scale of consciousness for nonhuman primates: A Delphi study
Sun, W., Xiangli, D., Yu, G., Yang, Y., He, B., Wei, Y., Li, S., Feng, Z., & Ma, C. (2023) · Science Progress · doi:10.1177/00368504231200995 · Cited by 6
The role of part-time arrangements in the sustainability of midwifery continuity of care models in Australia: An integrative review
Aleshin, O., & Donnellan-Fernandez, R. E. (2023) · European Journal of Midwifery · doi:10.18332/ejm/171359 · Cited by 5
Vocal Characteristics of Infants at Risk for Speech Motor Involvement: A Scoping Review
Long, H. L., Christensen, L., Hayes, S., & Hustad, K. C. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-23-00336 · Cited by 5
Duplicate references in the 'Introduction' and 'Discussion' sections of scientific articles on physical education and sports
Подрігало, Л., & Єрмаков, С. (2023) · Pedagogy of Health · doi:10.15561/health.2023.0203 · Cited by 5
Building Bridges between People with Stroke, Families, and Health Professionals: Development of a Blended Care Program for Self-Management
Pereira, C. M., Matos, M., Carvalho, D., Macedo, P., Calheiros, J. M., Alves, J., Ferreira, L. P., Dias, T. L., Madeira, R. N., & Jones, F. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13010300 · Cited by 6
Healthcare workers’ experience of a coach-led digital platform for better well-being
O’Donovan, R., Loughnane, C., Donnelly, J., Kelly, R., Kemp, D., McCarthy, L., Offiah, G., Sweeney, A., Duggan, A. P., & Dunne, P. J. (2024) · Coaching An International Journal of Theory Research and Practice · doi:10.1080/17521882.2024.2304793 · Cited by 5
“Analysis of the RCSI Coach Connect project followed a mixed methods research design in accordance with the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2019).”
Experiences and Views of Young People and Health Care Professionals of Using Social Media to Self-Manage Type 1 Diabetes Mellitus: Thematic Synthesis of Qualitative Studies
Ma, Y., Law, K., Hassan, L., Nenadić, G., & van der Veer, S. N. (2024) · JMIR Pediatrics and Parenting · doi:10.2196/56919 · Cited by 4
Getting to effective housing policy for health: a thematic synthesis of policy development and implementation
Nix, E., Ibbetson, A., Zhou, K., Davies, M., Wilkinson, P., Ludolph, R. A., & Pineo, H. (2024) · Cities & Health · doi:10.1080/23748834.2024.2328951 · Cited by 8
“Included empirical studies were appraised using the Mixed-Method Appraisal Tool (MMAT) for quantitative, qualitative, and mixed-methods studies (Hong et al .”
Method
Interventions to promote medication adherence among children with epilepsy: An integrative review
Kangwal, C., Thato, R., Ua-Kit, N., & Visudtibhan, A. (2024) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2024.06.015 · Cited by 5
A Systematic Review of Palliative Care Needs in Young-Onset Dementia
Dang, J., Yong, A. C. W., Fong, Z. H., Ang, K., & Ng, A. S. L. (2024) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2024.105219 · Cited by 6
Interactive Conversational Agents for Cigarette-Smoking and Vaping Cessation: A Mixed-Methods Systematic Review
Lachance, A., Da, S. M. A. R., Yameogo, A. R., Plaisimond, J., Naye, F., Yousefi, F., Bergeron, F., Sasseville, M., & Gagnon, M. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8110101 · Cited by 5
Self-citation
Treatment Strategies for Intermediate Spinal Instability Neoplastic Score Patients: A Systematic Review
Okai, B. K., Quiceno, E., Soliman, M. A. R., Francois, H. B., Khan, A., Roy, J. M., Levy, H. W., Aguirre, A. O., Pollina, J., & Mullin, J. P. (2025) · World Neurosurgery · doi:10.1016/j.wneu.2024.123627 · Cited by 5
Peer support for adult social care in prisons in England and Wales: a mixed-methods rapid evaluation
Walton, H., Massou, E., Sherlaw‐Johnson, C., Gipson, D., Wainwright, L., Harriott, P., Ng, P. L., Riley, S., Stephen Morris, A., & Fulop, N. (2025) · Health and Social Care Delivery Research · doi:10.3310/mwfd6890 · Cited by 6
The impact of teachers’ qualifications on development outcomes in early childhood: a systematic literature review
Yang, X., Rahman, M. N. A., & Sun, Y. (2025) · International Journal of Early Years Education · doi:10.1080/09669760.2025.2451301 · Cited by 5
“For this systematic review, the mixed methods appraisal tool (MMAT) was developed by Hong et al. (2019), which was utilised to evaluate the quality of the selected articles.”
Advocating for a Loved One in the Setting of Uncertainty
Umberger, R. A., Todt, K., Talbott, E., Sparks, L., & Thomas, S. P. (2020) · Dimensions of Critical Care Nursing · doi:10.1097/dcc.0000000000000449 · Cited by 5
“The family caregiver may receive notices about their possible discharge and may be engaged in discharge planning and preparation needed for their transition to home.(46) Although hospital administrators may consider that this is a standard practice, culture change may be needed to ensure consistent and family-centered…”
BackgroundMethod
Validation of the French version of the Multiple Sclerosis Intimacy and Sexuality Questionnaire 15 Tools which help nurse for assessing the effect of perceived multiple sclerosis symptoms on sexual activity and satisfaction
Lefebvre, S., Houot, M., Delgadillo, D., Dupart, M. D. C., Varin, D., Papeix, C., Sevin, M., Bourmaleau, J., Laigle–Donadey, F., & Jović, L. (2022) · Nursing Open · doi:10.1002/nop2.1323 · Cited by 5
A mixed-methods study of emotional support for families of organ donors in Hunan Province, China
Xie, W., Kong, S., He, H., Xiong, H., Zhu, Q., & Huang, P. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.952524 · Cited by 5
“an in-depth understanding of various interesting phenomena that cannot be fully understood using only a quantitative or a qualitative method (Schoonenboom and Johnson, 2017; Hong et al., 2019).”
Background
Development of a valid and reliable software customization model for SaaS quality through iterative method: perspectives from academia
Ali, A. Q., Sultan, A. B. M., Ghani, A., & Zulzalil, H. B. (2020) · PeerJ Computer Science · doi:10.7717/peerj-cs.294 · Cited by 6
Factors influencing medication adherence among patients with hypertension: a systematic reviewPreprint
Win, T., & Ruaisungnoen, S. B. A. W. (2021) · doi:10.17605/osf.io/zbc7u · Cited by 5
MAPPING KNOWLEDGE IN SCOPING REVIEWS: A GUIDE FOR NURSING RESEARCHERS AND ACADEMICS
Silva, A., Silva, V. S. E., Padilha, M. I., Petry, S., Mendes, K. D. S., & Costa, I. C. P. (2024) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2024-0074en · Cited by 4
Atherectomy and Intravascular Lithotripsy Provide Comparable Outcomes for Common Femoral Artery Atherosclerotic Disease: A Systematic Review With Meta-analysis
Bouchareine, N., Daniel, G., Ramos‐Pascual, S., Michalewska, K., Grew, B., Saffarini, M., Delannoy, P.-J., & Montveneur, M. (2025) · Journal of Endovascular Therapy · doi:10.1177/15266028251335174 · Cited by 4
“…quality of the eligible studies was assessed by 2 researchers (K.M. and B.G.) according to the Mixed Methods Appraisal Tool (MMAT), version 2018, 35 which consists of 7 questions: 2 general questions and 5 questions dependent on study design (quantitative nonrandomized or quantitative…”
In What Contexts, How, and Why Do Transition to Adulthood Interventions Work for Adolescents Living with Disabilities: A Realist Review
Zidan, A., Gabay, D., Poirier, S.-È., Grandisson, M., Jacob, S., Ouimet, M., Julien‐Gauthier, F., Martin-Roy, S., Lamontagne, M., Desmarais, C., et al. (2025) · Journal of Vocational Rehabilitation · doi:10.1177/10522263251356223 · Cited by 5
“The methodological quality of all included studies was appraised using the Mixed Methods Appraisal Tool (MMAT), version 2018 (Hong et al., 2019).”
Public health surveillance of tropical diseases in Madagascar: a scoping review of population burden, intervention strategies, and health system responses
Rajaofera, M. J. N., Liu, W., Tomboanona, S., Reziky, A. M., Kuang, D., & Xia, Q. (2025) · BMC Public Health · doi:10.1186/s12889-025-23802-0 · Cited by 4
Electronic patient-reported outcome systems and capabilities in cancer care: a systematic review
Salmani, H., Nasiri, S., Alemrajabi, M., & Ahmadi, M. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1560533 · Cited by 4
Artificial intelligence in nursing support for patients: A rapid review
Ito, Y., Kajiwara, K., Kako, J., Kobayashi, M., Tsubaki, M., Sakuramoto, H., Yamanaka, M., & Kakeda, T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106087 · Cited by 5
Leadership Roles, Care Models, and Effectiveness of Advanced Practice Nurse–Led Multidisciplinary Teams: A Scoping Review
Yang, J. A., Huang, J., Han, M., Qi, H., & Xu, M. (2025) · International Nursing Review · doi:10.1111/inr.70079 · Cited by 5
Application of the objective structured clinical examination in undergraduate nursing students: A systematic review with meta-synthesis
Garrido-Bueno, M., Romero‐Castillo, R., & Pabón‐Carrasco, M. (2025) · Teaching and learning in nursing · doi:10.1016/j.teln.2025.08.024 · Cited by 5
Mindfulness in Adaptation to Bereavement: A Systematic Review
Sun, X., Schroevers, M. J., Lessing, M., Karsten, J., & Eisma, M. (2025) · Clinical Psychology and Psychotherapy · doi:10.1002/cpp.70198 · Cited by 4
“Two reviewers (XS and ML) independently assessed the methodological quality of the included studies using the Mixed Methods Appraisal Tool (MMAT, version 2018; Hong et al. 2018), a validated and reliable tool designed to evaluate different study designs (Hong et al. 2019; Souto et al. 2015).”
Interprofessional education in healthcare settings: are healthcare professionals translating learning into practice? An integrated mixed methods systematic review
Field, R., Palermo, C. E., Kellett, J., Bevitt, T., Lambert, K., & Bacon, R. (2026) · Journal of Interprofessional Care · doi:10.1080/13561820.2025.2609086 · Cited by 4
“Ratings were presented descriptively, without an overall score, in accordance with MMAT guidance (Hong et al., 2019).”
Influential
A systematic review of the scope and impact of rural primary healthcare innovations using digital health technology
MacAskill, W., Gill, P., Woloszczuk, C., Alam, K., Wallis, K. A., McGrail, M. R., Kondalsamy‐Chennakesavan, S., & Nasir, B. (2026) · BMJ Open · doi:10.1136/bmjopen-2024-097874 · Cited by 7
Teacher-Targeted Bullying: A Meta-Analysis of Student Bullying of Teachers.
Qiao, B.-X., Wang, Y.-R., Chen-Bouck, L., & Zaeske, L. (2026) · Trauma, Violence, & Abuse · doi:10.1177/15248380251408406 · Cited by 4
Logistics Practices to Reduce Food Loss in Sustainable Agri-Food Supply Chains: From Literature Review to Research Framework
Yu, P., Hamid, R. B. A., Osman, L. H., Liao, J., & Ni, C. (2026) · Agriculture · doi:10.3390/agriculture16050587 · Cited by 4
“For qualitative, quantitative descriptive, quantitative non-randomized, and mixed methods studies, the original MMAT 2018 criteria were applied [64].”
Ensuring technological inclusivity for students with visual impairments in higher education: A systematic literature review
Gidisu, P., Abroampa, W. K., & Nyatsikor, M. K. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.102780 · Cited by 5
Brain functional alteration and cognitive performance in cardiovascular diseases: a systematic review of fMRI studies
Humayra, S., Yahya, N. A., Ning, C. J., Mir, I. A., Mohamed, A. L., & Manan, H. A. (2024) · Frontiers in Neurology · doi:10.3389/fneur.2024.1425399 · Cited by 4
“The methodological quality of studies was critically appraised using MMAT (24), which is a valid and reliable appraisal tool for systematic reviews (27, 28).”
Method
The social and emotional wellbeing of children in out-of-home care: A scoping review exploring structural and social factors
Healy, K. E., Reeves, L. S., Boman, M., Kolesnikova, I., Povey, J., Venables, J., Baxter, J., & Thompson, K. (2024) · Children and Youth Services Review · doi:10.1016/j.childyouth.2024.107991 · Cited by 4
The Effects of Cultural Factors on Food Assistance After Disaster: A Scoping Review
Koçak, H., Karakuş, E., Çalışkan, C., Dağ, N., Kınık, K., & Güzel, A. (2025) · Disaster Medicine and Public Health Preparedness · doi:10.1017/dmp.2025.1 · Cited by 4
Charting critical paths: Exploring how concept mapping amplifies nursing students’ critical thinking and elevates patient care – A scoping review
Thapa, M., Devi, S., & Mishra, R. (2025) · Journal of Education and Health Promotion · doi:10.4103/jehp.jehp_644_24 · Cited by 4
“Recognizing the lack of a specific checklist for mixed‑methods studies in SUMARI, the Mixed Methods Appraisal Tool (MMAT) [40] was used to assess the quality of studies with mixed‑methods design.”
Effectiveness of Virtual Reality Intervention in Enhancing Motor and Cognitive Functions in Down Syndrome: A Systematic Review
Baladaniya, M., & Baldania, S. (2025) · Cureus · doi:10.7759/cureus.80148 · Cited by 5
“Based on the various parameters described in the tool, the studies were graded as high (≥4 scores), low (≤1 score), or moderate quality (2-3 scores) [32].”
Influential
Impact of stigma on the HIV care cascade among older persons living with HIV in sub-Saharan Africa: protocol for a systematic review
Wandera, S. O., Nangendo, J., Kinengyere, A. A., Ocan, M., Katahoire, A., Semitala, F. C., Bogart, L. M., & Kamya, M. R. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-091718 · Cited by 4
“51 For randomised trials, we will use the revised Cochrane ‘Risk of bias’ tool for randomised trials (RoB V.2.0).”
Influential
Factors influencing health-promoting lifestyle among medical personnel: a systematic review protocol
Huang, P., Abai, D. S. A., Xiao, H., Zhang, Q., Xian, Z., & Abdullah, K. B. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-097470 · Cited by 4
Applications of schema therapy in young people: a systematic review
Joshua, P. R., Lewis, V., Kelty, S. F., & Boer, D. P. (2025) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2025.2522375 · Cited by 4
“The MMAT has high content validity although its discriminatory validity and inter-rater reliability are yet to be determined (Hong et al., 2019).”
Academic competition-based learning cultivates scientific literacy to promote professional competitiveness in medical undergraduates
Ruan, Y. H., Zhang, J., Wang, J., Jian, R., Mei, F., Li, H., Zhang, Y., Hu, Q., Xiao, L., Yang, Y. T., et al. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1590832 · Cited by 5
“The research content and activity mode of ACBL were constructed by a two-round modified Delphi study (24).”
Psychosocial burden of diabetes in Africa: A systematic review and meta-analysis
Boakye, M. D. S., Osei, E. P., Akyirem, S., Dare, S., Omololu, S. O., Ekpor, E., Bossman, I. F., Addo‐Mensah, D., Duodu, P. A., & Suglo, J. N. (2025) · Journal of Health Psychology · doi:10.1177/13591053251341786 · Cited by 5
“The quality of included studies was assessed using the Mixed Method Appraisal Tool (MMAT; Hong et al., 2019).”
Information needs and mHealth applications for carers of people with dementia in managing behavioural and psychological symptoms of care recipients: an integrative review protocol
Jagoda, T., Rathnayake, S., & Dharmaratne, S. D. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060414 · Cited by 4
Spiritual reading influenced by colors in the architecture of mosques based on the movement path of the worshiper (Case study: Nasir Al-Molk Mosque in Shiraz)
Ghochani, M., & Mousavi, S. M. (2022) · Haft Hesar Journal of Environmental Studies · doi:10.52547/hafthesar.11.39.6 · Cited by 4
Telesupervision in counselling and psychotherapy: a systematic review of the literature
Ratcliffe, K., & Kaluževičiūtė, G. (2023) · British Journal of Guidance and Counselling · doi:10.1080/03069885.2023.2264480 · Cited by 4
“…by Noyes et al. (2018), were used to evaluate quantitative and qualitative studies and Mixed Methods Appraisal Tools (MMAT) were used for mixed methods studies, to facilitate analysis of both quantitative and qualitative elements and consider this approach ’ s contribution (Hong et al., 2019).”
Desarrollo y validación de estándares para unidades de calidad de centros sanitarios
Aloy-Duch, A., Santiñà, M., Ramos-D’Angelo, F., Calo, L. A., Llaneza-Velasco, M. E., Fortuny-Organs, B., & Apezetxea-Celaya, A. (2023) · Journal of Healthcare Quality Research · doi:10.1016/j.jhqr.2023.09.009 · Cited by 4
Palliative Extubation in Pediatric Patients in the Intensive Care Unit and at Home: A Scoping Review
Neto, J., Casimiro, H. J., & Reis-Pina, P. (2023) · International Journal of Pediatrics · doi:10.1155/2023/6697347 · Cited by 4
Development of the Parental Experience with Care for Children with Serious Illnesses (PRECIOUS) quality of care measure
Ang, F. J. L., Gandhi, M., Østbye, T., Malhotra, C., Malhotra, R., Chong, P. H., Amin, Z., Chow, C., Tan, T. S. Z., Tewani, K. G., et al. (2024) · BMC Palliative Care · doi:10.1186/s12904-024-01401-x · Cited by 5
“…modified online Delphi approach to determine the content validity of the drafted items [28], which has been similarly used to determine the content validity of items in adult patient-reported experience measures [29], quality appraisal tools [30], and surveys measuring evidence-based practice [31].”
Assessing cognitive communication skills for returning to work following acquired brain injury: a systematic scoping review
Cameron, K., Cornwell, P., Watter, K., McLennan, V., & Wegener, N. (2024) · Disability and Rehabilitation · doi:10.1080/09638288.2024.2356683 · Cited by 4
A Systematic Literature Review of Strategies Implemented in Extended Education Settings to Address Children’s Mental Health and Wellbeing
Murray, S., March, S., Pillay, Y., & Senyard, E. (2024) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-024-00494-3 · Cited by 7
“Each version of the MMAT has undergone validity testing for interrater reliability, efficacy, and content validity, with the most recent showing excellent interrater agreement ( d = 0.87–1.0) on the items (Hong et al., 2019).”
MethodInfluential
The Caring Intensively Study: Three-Year Follow-Up Findings From a Mixed Methods Study of Children’s Psychological and Behavioral Responses After PICU Hospitalization
Rennick, J. E., Dougherty, G. E., Dryden‐Palmer, K., Campbell-Yeo, M. L., Knox, A. M., Chambers, C. T., Stack, D. M., Treherne, S. C., & Stremler, R. (2025) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003701 · Cited by 4
Patient and clinician characteristics and preferences for increasing participation in placebo surgery trials: a scoping review of attributes to inform a discrete choice experiment
Hinwood, M., Wall, L., Lang, D., Balogh, Z. J., Smith, A., Dowsey, M. M., Clarke, P., Choong, P., Bunzli, S., & Paolucci, F. (2022) · Trials · doi:10.1186/s13063-022-06277-x · Cited by 4
“For quantitative, qualitative, or mixed-methods studies utilising surveys, interviews, focus groups, or DCEs, we evaluated study quality using the Mixed Methods Appraisal Tool (MMAT), version 2018 [25, 29].”
Method
Sexual and reproductive health education for young people in Victoria, Australia: a mixed methods study
Walker, R., Mangelsdorf, R., Leahy, D., & Boyle, J. A. (2022) · Sex Education · doi:10.1080/14681811.2022.2103110 · Cited by 4
“The design and reporting of this study were guided by the Mixed Methods Assessment Tool (Hong et al. 2019).”
Method
Monitoring of high-risk children in health services: A geospatial mixed-methods study
Shibukawa, B. M. C., Uema, R. T. B., de Oliveira, N. N., de Oliveira, R. R., dos Santos, J. L. G., & Higarashi, I. H. (2022) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.5806.3777 · Cited by 4
Are principles of recovery-oriented practice evident in staff and service user perspectives on seclusion?
Stíobhairt, A., Staunton, D., & Guérin, S. (2023) · Mental Health Review Journal · doi:10.1108/mhrj-02-2022-0009 · Cited by 4
Discharge interventions for First Nations people with a chronic condition or injury: a systematic review
Coombes, J., Holland, A. J. A., Ryder, C., Finlay, S. M., Hunter, K., Bennett‐Brook, K., Orcher, P., Scarcella, M., Briscoe, K., Forbes, D., et al. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09567-5 · Cited by 4
Outcomes of usage of psychedelics by people reporting an eating disorder in clinical and non-clinical settings: a scoping review
Valdiviezo‐Oña, J., Toscano-Molina, L., Chávez, J. F., Criollo, J. E. H., & Paz, C. (2023) · Advances in Mental Health · doi:10.1080/18387357.2023.2235030 · Cited by 4
“This tool allows for the evaluation of the most common methodologies and designs with speci ﬁ c criteria for qualitative, quantitative randomised controlled trials, quantitative non-randomised, quantitative descriptive, and mixed methods studies (Hong et al., 2019).”
Impact of Standardised Patient Simulation Training on Clinical Competence, Knowledge, and Attitudes in Mental Health Nursing Education
Rabie, A. H., & Hakami, A. (2023) · Global Journal of Health Science · doi:10.5539/gjhs.v15n9p1 · Cited by 4
“An important step in systematic reviews is to critically assess the selected studies for their credibility, validity and reliability (Hong et al., 2019).”
BackgroundMethod
Play as a Stress-Coping Method Among Children in Light of the COVID-19 Pandemic: A Review
Raudenská, J., Gumančík, J., Raudenský, M., Pasqualucci, A., Tamayo, M. A. N., Varrassi, G., & Javůrková, A. (2023) · Cureus · doi:10.7759/cureus.43550 · Cited by 6
Content Validity-Based Evaluation Criteria System for Siting Wind-Solar Plants
Sachit, M. S., Shafri, H. Z. M., Abdullah, A. F., Rafie, A. S. M., & Hanoon, S. K. (2024) · Iraqi Journal of Science · doi:10.24996/ijs.2024.65.1.44 · Cited by 4
“The Delphi technique, used to converge the opinions of a group of experts, was followed over several rounds to collect ratings [42].”
Method
Health education programs for improving men’s engagement with health services in low- to middle-income countries: a scoping review protocol
Dzinamarira, T., Kuupiel, D., & Mashamba-Thompson, T. P. (2020) · Systematic Reviews · doi:10.1186/s13643-019-1266-3 · Cited by 4
HPV vaccination and Native Americans: protocol for a systematic review of factors associated with HPV vaccine uptake among American Indians and Alaska Natives in the USA
Gopalani, S. V., Sedani, A. E., Janitz, A. E., Clifton, S. C., Stoner, J. A., Peck, J. D., Comiford, A. L., Salvatore, A. L., & Campbell, J. E. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-035658 · Cited by 4
Implementation of online psychosocial interventions for people with neurological conditions and their caregivers: A systematic review protocol
Miao, M., Power, E., Rietdijk, R., Brunner, M., & Togher, L. (2021) · Digital Health · doi:10.1177/20552076211035988 · Cited by 4
The Effect of Community-Based Exercise on Health Outcomes for Indigenous Peoples with Type 2 Diabetes: A Systematic Review
Hurst, L., Kirwan, M., Christie, V., Cross, C., Baylis, S., White, L., & Gwynne, K. G. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21030290 · Cited by 5
A mixed‐method systematic review of the perspectives of young people, carers and professionals on psychiatric diagnosis in childhood and adolescence
Gill, J., Jones, A., Price, K., Goodison, E., & Tyson, P. J. (2024) · Journal of Child and Adolescent Psychiatric Nursing · doi:10.1111/jcap.12459 · Cited by 2
The effect of mindfulness-based interventions on stress and mental health outcomes among police officers: a systematic review and meta-analysis
Gaskin, J., Li, W. W., van Eijk, L., & Sarnyai, Z. (2025) · Police Practice and Research · doi:10.1080/15614263.2025.2506647 · Cited by 8
“The second stage of selection involved the methodological appraisal of the full text of each included article utilising the Mixed Methods Appraisal Tool (MMAT) Version 18 (Hong et al., 2019).”
Developing effective transition programmes for first-line nurse managers: A scoping review of evidence, barriers, and best practices
Esquisábel‐Soteras, B., Olano‐Lizarraga, M., Pardavila‐Belio, M. I., & Vázquez‐Calatayud, M. (2025) · Nurse Education in Practice · doi:10.1016/j.nepr.2025.104537 · Cited by 5
Exploring the trauma experiences of people working in homelessness: mixed-methods systematic review
Camp, B., Elahi, A., Waqas, A., & Weatherhead, S. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-087270 · Cited by 3
“⇒ The use of the Mixed Methods Appraisal Tool for quality appraisal and GRADE-CERQual for assessing confidence in qualitative findings strengthened the methodological rigour.”
Influential
Mapping the Comparative Effectiveness of Modified Constraint-Induced Movement Therapy, Hand-Arm Bimanual Intensive Training, and Mirror Therapy for Upper Extremity Rehabilitation in Children With Hemiplegic Cerebral Palsy: A Scoping Review
Bisen, R., & Ganvir, S. (2025) · Cureus · doi:10.7759/cureus.98440 · Cited by 3
Patient and clinician perspectives on the management of obesity in kidney failure prior to kidney transplantation: a mixed-methods systematic review
Oganesova, Z., MacLaughlin, H. L., McCafferty, K., Potthoff, S., Greenwood, S. A., Vickerstaff, V., Batterham, R. L., Afuwape, S. A., Motallebzadeh, R., & Brown, A. C. (2025) · EClinicalMedicine · doi:10.1016/j.eclinm.2025.103649 · Cited by 4
Psychosocial aspects of release in youth and developing athletes: A systematic review
McGlinchey, T. R., Wood, N. S., Saward, C., Healy, L. C., & Sarkar, M. (2025) · Journal of Sports Sciences · doi:10.1080/02640414.2025.2604953 · Cited by 3
“The quality of each included study was assessed using the mixed methods appraisal tool (MMAT; Hong et al., 2019).”
Impact of Vulvodynia on the Quality of Life of Women: A Rapid Review
Escajadillo, M. F. C., Córdoba, M. G. D. Q., Ramon, M., Gómez‐Cantarino, S., Romero-Arana, A., & Arroyo-Bello, E. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm15010070 · Cited by 3
“The evaluation of methodological quality and risk of bias, performed using the Mixed Methods Appraisal Tool (MMAT, version 2018) [15], revealed that the overall quality of the included studies was satisfactory.”
Bridging the Digital Divide in Diabetes Care: An Integrative Review of Barriers and Facilitators to Technology Use
Osei, E. P., Campos, S., & Martyn-Nemeth, P. A. (2025) · The Science of Diabetes Self-Management and Care · doi:10.1177/26350106251397917 · Cited by 3
Compulsive Media Consumption and Secondary Traumatization: A Systematic Review of Behavioral Addiction Mechanisms in Disaster-Related Media Exposure
Kınık, Ç., & Küçükali, A. S. (2025) · Uluslararası Davranış Sürdürülebilirlik ve Yönetim Dergisi · doi:10.71444/jobesam.1734513 · Cited by 3
Wearable Technologies in School Physical Education: A Systematic Review of Pedagogical Implications
Zhao, Z., & Su, Y. (2026) · Physical Education Theory and Methodology · doi:10.17309/tmfv.2026.1.07 · Cited by 7
“Using MMAT adds methodological transparency and rigour to this review (Hong et al., 2019).”
Barriers and facilitators of tuberculosis treatment adherence among nomadic populations in Sub-Saharan Africa: A scoping review protocol
Troop, P., Keam, B., Mullen, C., & Odhiambo, J. N. (2026) · PLoS ONE · doi:10.1371/journal.pone.0340307 · Cited by 3
Developmental care for preterm infants: a scoping review of interventions, outcomes, and implementation contexts
Hastuti, D., -, A., & Zubaidah, Z. (2026) · Frontiers in Pediatrics · doi:10.3389/fped.2026.1730571 · Cited by 3
Barriers and Enablers to Seeking Eye Care Among the Adult Indian Population: A Systematic Review
Yelagondula, V. K., Marmamula, S., Subramanian, A., & Lawrenson, J. G. (2026) · Ophthalmic and Physiological Optics · doi:10.1007/s44402-026-00027-1 · Cited by 3
Suicide and self-harm prevention on social media: A systematic review of interventions and user-led practices
Cooper, C., Sala, L. L., Patlamazoglou, L., Patrick, P., Sabo, A., Rajaram, G., & Robinson, J. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101067 · Cited by 4
Sustainable Anesthesia: A Scoping Review of Carbon Reduction Strategies
Moser, A. J., Hammer, S., König, A., Bornemann‐Cimenti, H., Heuschneider, L., & Lang-Illievich, K. (2026) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000008046 · Cited by 3
Artificial Intelligence in Physical, Occupational and Neuro-Rehabilitation: Clinical Effectiveness, Prognostic Performance, and Pre-Implementation Feasibility – A Systematic Review
Arab, R. A. E., Moosa, O. A. A., Almagharbeh, W. T., Abuadas, F. H. A., Abdalla, N., Abdrbo, A. A., & Hassanein, S. E. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02400-6 · Cited by 3
“…synthesis of the primary evidence base; undertaking methodological and model-readiness appraisal using Mixed Methods Appraisal Tool (MMAT) [15, 16 ] and PROBAST + AI [ 17], where applicable; and deriving practice-oriented recommendations from direct assessment of trial conduct, external…”
Prevalencia de miopía, hipermetropía y astigmatismo en México: Una revisión sistemática
Ortiz, M. I., Campuzano-Revilla, G. P., Muñoz‐Pérez, V. M., & Cuevas‐Suárez, C. E. (2022) · Educación y Salud Boletín Científico Instituto de Ciencias de la Salud Universidad Autónoma del Estado de Hidalgo · doi:10.29057/icsa.v10i20.8591 · Cited by 3
“Evaluación de la calidad metodológica La evaluación de la calidad se realizó mediante la herramienta Mix Methods Appraisal Tool (MMAT).(14) Los estudios se analizaron en los siguientes dominios: preguntas de investigación claras, recolección de datos adecuada, estrategias de muestreo relevantes, muestras representativ…”
Background
Nursing practices on sexual health for people with neurological disorders: A narrative review
Lefebvre, S., Cousin, L., Bourmaleau, J., Rohé, S., Devictor, J., Martin, P., & Jović, L. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17318 · Cited by 3
Clinicians' Barriers to Screening and Diagnosing Diabetes Distress in Patients with Type 1 and 2 Diabetes Mellitus: A systematic review
Nhlabatsi, M. L., van Teijlingen, E., & Hundley, V. A. (2024) · African Journal of Health Sciences · doi:10.4314/ajhs.v36i5.2 · Cited by 3
“These appraisal tools provided a structural framework for assessing the rigour of selected studies (53, 54).”
Method
What influences the implementation of clinical guidelines related to enteral nutrition in the intensive care unit: A mixed‐methods systematic review
Li, J. Q., Sun, T., Zuo, J. T., Yao, X., Cai, L. N., Zheng, H. Y., & Ye, X. H. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16384 · Cited by 4

Authors: Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
Affiliations: McGill University (CA)
Universitat Oberta de Catalunya (ES)
University of Warwick (GB)
University of Canberra (AU)
Université de Sherbrooke (CA)
Université Laval (CA)
University of Sheffield (GB)
Institut National de la Recherche Scientifique (CA)
Corresponding author: Pluye, Pierre
ORCID: Hong, Quan Nha
Pluye, Pierre
Fàbregues, Sergi
Bartlett, Gillian
Boardman, Felicity
Cargo, Margaret
Dagenais, Pierre
Gagnon, Marie-Pierre
Griffiths, Frances
Nicolau, Belinda
O’Cathain, Alicia
Rousseau, Marie-Claude
Vedel, Isabelle
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2019-07
Volume: 111
Pages: 49-59.e1
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2019.03.008
OpenAlex: W2924322406
Cited by: 1,082
References: 50
Citations by year: 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
FWCI: 168.24
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by-nc-nd
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
Locations: Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
PubMed (publishedVersion)
The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
HAL (submittedVersion, cc-by-nc-nd)
eScholarship@McGill (McGill) (submittedVersion)
EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
Warwick Research Archive Portal (University of Warwick) (submittedVersion)
Topics: Social Sciences › Sociology and Political Science › Delphi Technique in Research
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Health Sciences › Nursing › Research and Theory › Health Education and Validation
Keywords: Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
MeSH terms: Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks