Cited by 1,082Delphi Technique in ResearchMeta-analysis and systematic reviewsHealth Education and Validation

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

Open access

Hong, Quan Nha, Pluye, Pierre, Fàbregues, Sergi, Bartlett, Gillian, Boardman, Felicity, Cargo, Margaret, Dagenais, Pierre, Gagnon, Marie-Pierre, Griffiths, Frances, Nicolau, Belinda, O’Cathain, Alicia, Rousseau, Marie-Claude, Vedel, Isabelle·2019·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2019.03.008

Open this paper in CoChat — free account Open PDF

Abstract

OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2019.03.008

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
Method
Achieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
Background
Determination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
Background
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
Method
Is the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
Background
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
Method
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657

Show all 50 references

Sorted by citations

Items 901–1,000 of 1,001

Additional file 1 of Patient and clinician characteristics and preferences for increasing participation in placebo surgery trials: a scoping review of attributes to inform a discrete choice experiment
Hinwood, M., Wall, L., Lang, D., Balogh, Z. J., Smith, A., Dowsey, M., Clarke, P., Choong, P., Bunzli, S., & Paolucci, F. (2022) · Figshare · doi:10.6084/m9.figshare.19589151
Additional file 4 of Factors associated with short birth interval in low- and middle-income countries: a systematic review
Pimentel, J., Ansari, U., Omer, K., Gidado, Y., Baba, M. C., Andersson, N., & Cockcroft, A. (2020) · Figshare · doi:10.6084/m9.figshare.11978673
Additional file 1 of Determinants of implementation for group medical visits for patients with chronic pain: a systematic review
Roth, I., Tiedt, M., Brintz, C., Thompson‐Lastad, A., Ferguson, G. M., Agha, E., Holcomb, J. L., Gardiner, P. M., & Leeman, J. (2024) · Figshare · doi:10.6084/m9.figshare.25893759
Additional file 1 of An examination of the psychosocial consequences experienced by children and adolescents living with congenital heart disease and their primary caregivers: a scoping review protocol
Dorfman, T. L., Archibald, M., Haykowsky, M., & Scott, S. D. (2023) · Figshare · doi:10.6084/m9.figshare.23290460
Additional file 1 of Psychological factors associated with foot and ankle pain: a mixed methods systematic review
Cotchett, M., Frescos, N., Whittaker, G. A., & Bonanno, D. R. (2022) · Figshare · doi:10.6084/m9.figshare.19120337
Additional file 1 of Care programs and their components for patients with idiopathic pulmonary fibrosis: a systematic review
Delameillieure, A., Vandekerkhof, S., Grootven, B. V., Wuyts, W. A., & Dobbels, F. (2021) · Figshare · doi:10.6084/m9.figshare.15179603
Additional file 3 of Barriers and facilitators of access to maternity care for African-born women living in Australia: a meta-synthesis of qualitative evidence
Bali, A. G., Vasilevski, V., & Sweet, L. P. (2024) · Figshare · doi:10.6084/m9.figshare.26759033
Additional file 6 of The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Figshare · doi:10.6084/m9.figshare.26679638
Additional file 8 of An examination of the psychosocial consequences experienced by children and adolescents living with congenital heart disease and their primary caregivers: a scoping review protocol
Dorfman, T. L., Archibald, M., Haykowsky, M., & Scott, S. D. (2023) · Figshare · doi:10.6084/m9.figshare.23290481
Additional file 1 of What about the fundamentals of nursing—its interventions and its continuity among older people in need of home- or facility-based care: a scoping review
Nordaunet, O. M., Gjevjon, E. R., Olsson, C., Aagaard, H., & Borglin, G. (2024) · Figshare · doi:10.6084/m9.figshare.26670597
Additional file 1 of Interventions supporting cancer patients in making decisions regarding participation in clinical trials - a systematic review
Hillersdal, L., Nielsen, Z. E., Nørmark, A. T., Knoop, A. S., & Piil, K. (2022) · Figshare · doi:10.6084/m9.figshare.21405721
Additional file 4 of Development of the Parental Experience with Care for Children with Serious Illnesses (PRECIOUS) quality of care measure
Ang, F. J. L., Gandhi, M., Østbye, T., Malhotra, C., Malhotra, R., Chong, P. H., Amin, Z., Chow, C., Tan, T. S. Z., Tewani, K. G., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26690659
Additional file 1 of Teamwork and implementation of innovations in healthcare and human service settings: a systematic review
McGuier, E. A., Kolko, D. J., Aarons, G. A., Schachter, A., Klem, M. L., Diabes, M. A., Weingart, L. R., Salas, E., & Wolk, C. B. (2024) · Figshare · doi:10.6084/m9.figshare.26749415
Additional file 2 of Identifying determinants of varenicline adherence using the Theoretical Domains framework: a rapid review
Minian, N., Wong, M., Hafuth, S., Rodak, T., Rahimi, A., Gjomema, D., Rose, J., Zawertailo, L. A., Ratto, M., & Selby, P. (2024) · Figshare · doi:10.6084/m9.figshare.26688506
Additional file 1 of The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Figshare · doi:10.6084/m9.figshare.25203625
Additional file 2 of Usability and utility of eHealth for physical activity counselling in primary health care: a scoping review
Wattanapisit, A., Tuangratananon, T., & Wattanapisit, S. (2020) · Figshare · doi:10.6084/m9.figshare.13203420
Additional file 3 of Identifying determinants of varenicline adherence using the Theoretical Domains framework: a rapid review
Minian, N., Wong, M., Hafuth, S., Rodak, T., Rahimi, A., Gjomema, D., Rose, J., Zawertailo, L. A., Ratto, M., & Selby, P. (2024) · Figshare · doi:10.6084/m9.figshare.26688509
Additional file 3 of Health outcomes and implementation barriers and facilitators of comprehensive geriatric assessment in community settings: a systematic integrative review [PROSPERO registration no.: CRD42021229953]
Sum, G., Nicholas, S. O., Nai, Z. L., Ding, Y. Y., & Tan, W. S. (2022) · Figshare · doi:10.6084/m9.figshare.19687208
Additional file 1 of Mapping evidence on adolescents’ HIV-positive status disclosure in sub-Saharan Africa: a protocol for a scoping review
Adzordor, P., Avoka, C., Bawontuo, V., Agbesi, S., & Kuupiel, D. (2020) · Figshare · doi:10.6084/m9.figshare.13336153
Additional file 4 of A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine
Newington, L., Wells, M., Adonis, A., Bolton, L., Saghdaoui, L. B., Coffey, M., Crow, J., Fadeeva Costa, O., Hughes, C., Savage, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14516080
Additional file 3 of Care programs and their components for patients with idiopathic pulmonary fibrosis: a systematic review
Delameillieure, A., Vandekerkhof, S., Grootven, B. V., Wuyts, W. A., & Dobbels, F. (2021) · Figshare · doi:10.6084/m9.figshare.15179609
Additional file 2 of Using a theoretical framework to inform implementation of the patient-centred medical home (PCMH) model in primary care: protocol for a mixed-methods systematic review
Mazevska, D., Pearse, J., & Tierney, S. (2022) · Figshare · doi:10.6084/m9.figshare.21622272
Additional file 1 of Seeking connection: a mixed methods study of mental well-being and community volunteerism among international migrants in Japan
Miller, R., Ong, K. I. C., Choi, S., Shibanuma, A., & Jimba, M. (2020) · Figshare · doi:10.6084/m9.figshare.12839117
Additional file 2 of Care programs and their components for patients with idiopathic pulmonary fibrosis: a systematic review
Delameillieure, A., Vandekerkhof, S., Grootven, B. V., Wuyts, W. A., & Dobbels, F. (2021) · Figshare · doi:10.6084/m9.figshare.15179606
Additional file 4 of What about the fundamentals of nursing—its interventions and its continuity among older people in need of home- or facility-based care: a scoping review
Nordaunet, O. M., Gjevjon, E. R., Olsson, C., Aagaard, H., & Borglin, G. (2024) · Figshare · doi:10.6084/m9.figshare.26670606
Additional file 2 of Health outcomes and implementation barriers and facilitators of comprehensive geriatric assessment in community settings: a systematic integrative review [PROSPERO registration no.: CRD42021229953]
Sum, G., Nicholas, S. O., Nai, Z. L., Ding, Y. Y., & Tan, W. S. (2022) · Figshare · doi:10.6084/m9.figshare.19687205
Additional file 2 of Seeking connection: a mixed methods study of mental well-being and community volunteerism among international migrants in Japan
Miller, R., Ong, K. I. C., Choi, S., Shibanuma, A., & Jimba, M. (2020) · Figshare · doi:10.6084/m9.figshare.12839120
Additional file 7 of An examination of the psychosocial consequences experienced by children and adolescents living with congenital heart disease and their primary caregivers: a scoping review protocol
Dorfman, T. L., Archibald, M., Haykowsky, M., & Scott, S. D. (2023) · Figshare · doi:10.6084/m9.figshare.23290478
Additional file 1 of The roles of pets in long-term care at home: a qualitative study
Reniers, P. W. A., Leontjevas, R., Declercq, I., Enders‐Slegers, M., Gerritsen, D. L., & Hediger, K. (2023) · Figshare · doi:10.6084/m9.figshare.24465049
Additional file 5 of Patient and clinician characteristics and preferences for increasing participation in placebo surgery trials: a scoping review of attributes to inform a discrete choice experiment
Hinwood, M., Wall, L., Lang, D., Balogh, Z. J., Smith, A., Dowsey, M., Clarke, P., Choong, P., Bunzli, S., & Paolucci, F. (2022) · Figshare · doi:10.6084/m9.figshare.19589172
Additional file 1 of Barriers and facilitators of access to maternity care for African-born women living in Australia: a meta-synthesis of qualitative evidence
Bali, A. G., Vasilevski, V., & Sweet, L. P. (2024) · Figshare · doi:10.6084/m9.figshare.26531026
Additional file 4 of An examination of the psychosocial consequences experienced by children and adolescents living with congenital heart disease and their primary caregivers: a scoping review protocol
Dorfman, T. L., Archibald, M., Haykowsky, M., & Scott, S. D. (2023) · Figshare · doi:10.6084/m9.figshare.23290469
Additional file 4 of Patient and clinician characteristics and preferences for increasing participation in placebo surgery trials: a scoping review of attributes to inform a discrete choice experiment
Hinwood, M., Wall, L., Lang, D., Balogh, Z. J., Smith, A., Dowsey, M., Clarke, P., Choong, P., Bunzli, S., & Paolucci, F. (2022) · Figshare · doi:10.6084/m9.figshare.19589166
Additional file 2 of Psychological factors associated with foot and ankle pain: a mixed methods systematic review
Cotchett, M., Frescos, N., Whittaker, G. A., & Bonanno, D. R. (2022) · Figshare · doi:10.6084/m9.figshare.19120340
Additional file 3 of The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Figshare · doi:10.6084/m9.figshare.26679629
Additional file 1 of Health outcomes and implementation barriers and facilitators of comprehensive geriatric assessment in community settings: a systematic integrative review [PROSPERO registration no.: CRD42021229953]
Sum, G., Nicholas, S. O., Nai, Z. L., Ding, Y. Y., & Tan, W. S. (2022) · Figshare · doi:10.6084/m9.figshare.19687202
Additional file 1 of Factors associated with short birth interval in low- and middle-income countries: a systematic review
Pimentel, J., Ansari, U., Omer, K., Gidado, Y., Baba, M. C., Andersson, N., & Cockcroft, A. (2020) · Figshare · doi:10.6084/m9.figshare.11978655
Additional file 2 of An examination of the psychosocial consequences experienced by children and adolescents living with congenital heart disease and their primary caregivers: a scoping review protocol
Dorfman, T. L., Archibald, M., Haykowsky, M., & Scott, S. D. (2023) · Figshare · doi:10.6084/m9.figshare.23290463
Additional file 3 of A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine
Newington, L., Wells, M., Adonis, A., Bolton, L., Saghdaoui, L. B., Coffey, M., Crow, J., Fadeeva Costa, O., Hughes, C., Savage, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14516077
Additional file 5 of Discharge interventions for First Nations people with a chronic condition or injury: a systematic review
Coombes, J., Holland, A. J. A., Ryder, C., Finlay, S. M., Hunter, K., Bennett‐Brook, K., Orcher, P., Scarcella, M., Briscoe, K., Forbes, D., et al. (2023) · Figshare · doi:10.6084/m9.figshare.23488937
Additional file 5 of The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Figshare · doi:10.6084/m9.figshare.26679635
Additional file 3 of What about the fundamentals of nursing—its interventions and its continuity among older people in need of home- or facility-based care: a scoping review
Nordaunet, O. M., Gjevjon, E. R., Olsson, C., Aagaard, H., & Borglin, G. (2024) · Figshare · doi:10.6084/m9.figshare.26670603
Additional file 2 of Barriers and facilitators of access to maternity care for African-born women living in Australia: a meta-synthesis of qualitative evidence
Bali, A. G., Vasilevski, V., & Sweet, L. P. (2024) · Figshare · doi:10.6084/m9.figshare.26531101
Validation of Tennis Talent Scouting Test
Nugroho, W., Sukamti, Budiarti, R., & Pranatahadi, S. (2025) · Proceedings in Technology Transfer · doi:10.1007/978-981-96-8249-2_24
Development Psychomotor Instrument of Coaching Game for Upgrading PM-515 at Primary School Long Jump Subject
Arga, M. W., Siswantoyo, & Yudanto (2025) · Proceedings in Technology Transfer · doi:10.1007/978-981-96-8249-2_23
Redefining social support: a scoping review of the effects of digital technologies on the social support of older workers
Bostan, C. M., Ševčíková, A., YAYLAGÜL, N. K., Kulla, G., Yıldızer, G., Mercan, M. A., Barlın, H., Gu, Y., Nilsson, K. O., Grad, D. A., et al. (2026) · BMC Public Health · doi:10.1186/s12889-025-26155-w
Teaching web design in secondary schools through HTML and CSS: A systematic synthesis of research evidence and pedagogical implications
Kharchenko, O. Y., & Semerikov, S. O. (2026) · Educational Dimension · doi:10.31812/ed.1061
Effects of dental implants on dietary and nutritional outcomes in older adults: a systematic review
Induijaa, S., Senevirathna, K., Batuwanthudawa, M., Kodikarage, S. C. H., Bandara, P., Bandara, D. L., Kanmodi, K. K., Jayasinghe, R. D., & Jayasinghe, R. M. (2026) · BMC Oral Health · doi:10.1186/s12903-026-07708-w
Digital Psychosocial Follow-Up for Survivors of Childhood Critical Illness: A Systematic Literature Review (Preprint)Preprint
Hagen, M. H., Jaccheri, L., & Hartvigsen, G. (2026) · doi:10.2196/preprints.89825
Kratom use in adolescents and college students: a systematic review of prevalence, patterns, and risk factors across community and rehabilitation settings
Govarthnapany, N., Dékány, L., Gabrhelík, R., & Singh, D. (2026) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990.2025.2609965
“Since these tools do not provide a numerical scale for assessing study quality (22, 23), all four authors collaboratively evaluated and discussed the overall quality of the studies.”
Mindfulness-Based Interventions Targeting Social Isolation and Loneliness for Refugee, Immigrant, and Migrant Populations: A Systematic Literature Review
Disney, L., Owen-Smith, A., Bray-Hayes, M., & Moon, Y. (2026) · Studies in Clinical Social Work Transforming Practice Education and Research · doi:10.1080/28376811.2026.2622967
Exploring the role of music listening in cancer care: A systematic literature review
Harrison, S. (2026) · South African Journal of Arts Therapies · doi:10.36615/je06za50
“The Mixed Methods Appraisal Tool (Hong et al., 2019) was employed for studies using mixed-methods designs, as it allows for a comprehensive and integrated assessment of the quality of both qualitative and quantitative components within a single framework.”
Unified tools for assessing the methodological quality of intervention effects in rapid reviews: a scoping reviewPreprint
Edwards, D. J., Clark, E., Csontos, J. K., Dobbins, M., Gillen, E., Hounsome, J. C., Neil‐Sztramko, S., Lewis, R., Mann, M. K., & Prue, G. (2026) · Research Square · doi:10.21203/rs.3.rs-8724550/v1
Interventions to Mitigate the Effects of Housing Insecurity on Child and Adolescent Health: A Scoping Review
Coy-Pérez, A., Carrere, J., Fernández, A., Borrell, C., Serral, G., Sánchez, E., Macaya, I., Vásquez-Vera, H., Vásquez-Vera, C., Mehdipanah, R., et al. (2026) · Public health reviews · doi:10.3389/phrs.2025.1609177 · Cited by 1
“For this purpose, we used the Mixed Methods Appraisal Tool (MMAT) [43], a validated and widely used instrument [44–46] designed to assess a range of study designs, including qualitative studies, randomized controlled trials, non-randomized quantitative studies, quantitative descriptive studies, and…”
Delphi method to determine weights of the meridians and acupoints for tapping therapy in frail older patients
Chen, Y., Gu, X., Lan, X., Wang, X., & Ren, X. (2026) · European Journal of Integrative Medicine · doi:10.1016/j.eujim.2026.102644
Receipt of Medicines Information From the Internet and Other Information Sources Among Adult Medicine Users in Developed Economies, 2010-2025: Systematic Review
Mononen, N., Salo, M.-M., Timonen, D., Pohjanoksa-Mäntylä, M., & Airaksinen, M. (2026) · Journal of Medical Internet Research · doi:10.2196/71984
Sense of Grip on Disease Among Persons Living with HIV: Insights from a Mixed-method Study in a Tertiary Care Hospital of Kolkata, India
Roy, S., Bandyopadhyay, L., Modak, D., Sahu, M., Ghosh, S., & Paul, B. (2026) · Indian Journal of Community Medicine · doi:10.4103/ijcm.ijcm_127_25
“[18] r esuLts”
Mapping Evidence on the Effectiveness of Mind-Body Interventions and Brain Gym on Cognitive Function in Older Adults: A Scoping Review
Sakpal, S., Ganvir, S., Suvarna, D., Ranade, P., & Pitchai, P. (2026) · Cureus · doi:10.7759/cureus.104687
The occupational well-being of nurse educators, its related challenges, stressors and their reduction in work life balance—A systematic mixed-methods study
Bhurtun, H. D., Kähkönen, O., Vauhkonen, A., Valaydum, N., & Saaranen, T. K. (2026) · Nurse Education Today · doi:10.1016/j.nedt.2026.107071
Impact of Mindful Eating Interventions on Food Intake and Eating Behaviors in Youth: A Systematic Review and Meta-Analysis
Lai, Y.-R., & Chang, Y.-C. (2026) · Mindfulness · doi:10.1007/s12671-026-02785-y
Informal Coercion Experienced by Adolescents in Mental Health Care—A Systematic Review
Överlund, T., & Lantta, T. (2026) · International Journal of Mental Health Nursing · doi:10.1111/inm.70245
“Each study was appraised using seven criteria, with response options of ‘Yes’, ‘No’, or ‘Can't tell’ (Hong et al. 2019).”
Evaluating evidence-based recruitment strategies for Alzheimer’s disease and related dementias clinical trial research: A literature review
Jacobson, M., Deuschle, C., Peneva, D., Wang, A. N.-Y., Roache, C. E., Walsh, M., Ferrell, P. B., Manetas, M.-A., Raman, R., Aisen, P. S. S., et al. (2026) · The Journal of Prevention of Alzheimer s Disease · doi:10.1016/j.tjpad.2026.100532 · Cited by 1
“…the MMAT developers provide a scoring algorithm that has been widely adopted in the literature, they recommend generating summaries of study quality that reflect the raw MMAT criteria ratings rather than relying solely on aggregated scores, which assign equal weight to all MMAT criteria [86].”
Attitudes and perceptions of the application of large language models among health professionals: A mixed-methods systematic review
Luo, W., Feng, T., Zhang, T., Chen, X., Lu, X., Li, Y., Hou, C., & Gao, J. (2026) · Public Health · doi:10.1016/j.puhe.2026.106252
Exploring the nexus of cross-cultural innovation and design entrepreneurship: a systematic review
Omar, W. N. F. W., Anuar, A., Hamid, A. H. A., Shaari, N., Ueda, A., & OMORI, K. (2026) · Journal of global entrepreneurship research · doi:10.1007/s40497-026-00513-3
Avoiding Fragmentation and Salami Slicing in Mixed Methods Research
Nijkamp, N., Calleja, P., Jack, L., & Sahay, A. (2026) · Clinical Nursing Research · doi:10.1177/10547738261431626
“Additionally, the MMAT is used mainly in literature reviews to critique the quality of mixed meth-ods studies (Hong et al., 2019).”
Unified tools for assessing the methodological quality of intervention effects in rapid reviews: a scoping reviewPreprint
Edwards, D. J., Clark, E., Csontos, J. K., Dobbins, M., Gillen, E., Hounsome, J. C., Neil-Sztramko, S. E., Lewis, R., Mann, M. K., & Prue, G. (2026) · Research Square · doi:10.21203/rs.3.rs-8724550/v2
Convergent and Discriminant Validity Evidence of the Methodological Quality Scale for Studies Based on Observational Methodology (MQSOM)
López-Arenas, D., Sanduvete‐Chaves, S., Mena-Raposo, J., & Moscoso, S. C. (2026) · The Spanish Journal of Psychology · doi:10.1017/sjp.2026.10021
Mental health practitioners’ experiences and practices in making decisions about onward care for patients presenting to emergency departments with self-harm or suicidal ideation: systematic review and meta-synthesis
Suzuki, M., Bakou, A. E., Dove, J., & McCabe, R. (2026) · BJPsych Open · doi:10.1192/bjo.2026.11007
Teaching resuscitation of critically ill patients to medical learners in the clinical setting – A scoping review
Turcotte, A.-S., Szulewski, A., McKeown, S., Hajjaj, O., Cowls, M. H., Morrison, C., Zhu, N., Pentland, V., Baril, L., Garceau-Tremblay, Z., et al. (2026) · Canadian Journal of Emergency Medicine · doi:10.1007/s43678-026-01150-1
Planning reliability in BIM-driven industrialized construction
León, W., Guevara, J., & Garcia-Lopez, N. P. (2026) · International Journal of Construction Management · doi:10.1080/15623599.2026.2643730
Le vécu de l'accouchement chez la femme : évaluation clinique, étude des facteurs de risque et de protection, et des conséquences sur l'état psychologique maternel, la relation conjugale et la relation mère-bébé
Chabbert, M. (2020) · theses.fr (ABES) · doi:10.70675/6a0d17e1zd4e1z4bcfz9a29zec72cb0995b7
Intelligence artificielle et formation des professionnels de la santé : une revue intégrative des apports, défis et enjeux de médiation
Naffi, N., Bahlouli, Y. E., Fortier, N., Doumbouya, M. B., Shakeraneh, S., Faraj, D. A., Grégoire, J., Beaulieu, N., & Whelan, K. (2026) · Médiations et médiatisations · doi:10.52358/mm.vi25.507
Technology-based solutions to alleviate loneliness in professional caregivers and the elderly with loss of autonomy
Zadeh, S. O. M. (2024) · theses.fr (ABES) · doi:10.70675/d866982azaf08z4499z8a48z9c8da202c81f
Psychotherapeutic Techniques for Dyspnea in Cancer and Chronic Illness: Systematic Review
Kako, J., Kajiwara, K., Kobayashi, M., Ito, Y., Ichikura, K., Matsuda, Y., & Yamaguchi, T. (2026) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2026.03.021
Environmental NGO Advocacy and Governance Transformation in Mining Regions: A Systematic Review
Syam, R., Wisadirana, D., Susilo, E., & Nurhadi, I. (2026) · Environmental Quality Management · doi:10.1002/tqem.70352
Deep Learning Applications in STEM Education: A Systematic Review of Pedagogical Innovations in Numeracy and Science Learning
Sudiarti, M., & Harisman, Y. (2026) · Jurnal Penelitian Pendidikan IPA · doi:10.29303/jppipa.v12i3.14846
“…was evaluated using a modified version of the Mixed Methods Appraisal Tool (MMAT), which is designed to appraise the methodological quality of diverse study designs including quantitative, qualitative, and mixed-methods research (Brezovec et al., 2025; Burgess et al., 2025; Hong et al., 2019).”
The Role of Social Media in Dental Education: A Qualitative Study
Elraggal, A., Sims, S., & Louca, C. (2026) · European Journal Of Dental Education · doi:10.1111/eje.70167
Polyvinyl chloride-related insulin adsorption in critical care: a narrative review with implications for patient safety and green hospital reform
Elgazzar, S. H., Ibrahim, R. K., Ayoub, Y. M., Alanazi, R., Alanazi, A. K., Tantawi, H. R., Al-Qudimat, A. R., Alsharidah, A., & Hendy, A. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1718431
“Although quality assessment is not required for narrative reviews (19), a quality assessment of the research studies was undertaken using the Mixed Methods Appraisal Tool (MMAT) to enhance the overall rigor of the review process (20, 21).”
Contextual factors influencing the integration of physical activity policy, systems, and environmental interventions in the cooperative extension system: a systematic review
Palmer, S., Shaw, E., Grocke-Dewey, M., Corlew, K., Flournoy, L., & Balis, L. E. (2026) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-026-01927-8
“The Mixed Methods Appraisal Tool (MMAT) version 2018 [32] was used to appraise article quality.”
Research-Informed Design Principles in the Development of Professional Competency Frameworks: A Systematic Review
Abdullah, C. U., Rahmawati, S., Cotton, W., & Peralta, L. R. (2026) · Education Sciences · doi:10.3390/educsci16050725
The Impact of Test Preparation Practices on L2 Test Performance and Language Proficiency: A Systematic Review
Stansfield, L., Sénécal, A.-M., He, S., & Suvorov, R. (2026) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2026.2662635
Factors Influencing Job Satisfaction in Community Health Nursing: A Mixed Methods Review
Okullu, S., Kerr, L., Larkin, J., & Cithambaram, K. (2026) · Journal of Community Health Nursing · doi:10.1080/07370016.2026.2667174
“The MMAT is designed to assess the methodological quality of qualitative, quantitative, and mixed-methods studies (Hong et al., 2019).”
Mapping the Effectiveness of Compensatory Stepping Training for Balance and Fall Prevention in Community-Dwelling Older Adults: A Scoping Review
Bhattad, R., & Tungikar, S. L. (2026) · Cureus · doi:10.7759/cureus.108685
Access to clinical data for direct oral anticoagulant prescriptions in community pharmacy and implications for medication optimisation: a mixed-methods study
van Paassen, J. G., Deneer, V. H. M., Kempen, T. G. H., & Bouvy, M. L. (2026) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-026-02159-3
“We applied the Mixed Methods Appraisal Tool (MMAT) and the Consolidated criteria for reporting qualitative research (COREQ; Online Resource 1) to guide study design and reporting [23, 24].”
Systematic Review and Meta-Synthesis of Post-registration Education Program Evaluations for Pharmacists in Australia
Mcdonough, T., Babu, D., Page, A., Ellett, L. M. K., Etherton‐Beer, C., Borg, C., & Johnson, J. L. (2026) · American Journal of Pharmaceutical Education · doi:10.1016/j.ajpe.2026.102010
Leveraging Intelligence for Economic Transformation: The Role of National Intelligence Agencies in Fostering Gross Capital Formation
Maune, A. (2026) · Journal of Intelligence Studies in Business · doi:10.37380/jisib.v16i1.3135
Indicators, frameworks, and influencing factors for evaluating the performance of mental health programs in primary health care: a scoping review
Shojaei, S., Mahfoozpour‌, S., Asl, I. M., & Hessam, S. (2026) · BMC Health Services Research · doi:10.1186/s12913-026-14857-9
“…using tools appropriate to the study type: Quantitative studies were assessed using the Joanna Briggs Institute (JBI) tool [38], qualitative studies using the Critical Appraisal Skills Programme (CASP) checklist [39], and mixed-methods studies using the Mixed Meth-ods Appraisal Tool (MMAT) [40].”
Experiences of Inclusion and Exclusion Across the Gynaecological Cancer Continuum for Individuals With Diverse Gender and Sexuality Backgrounds: A Systematic Review
Boding, S., Webb, S. N., Butler, T., Hutchinson, A. O., Hanlon, L., & Russell, H. (2026) · Psycho-Oncology · doi:10.1002/pon.70518
Advancing agro-food processing productivity through blended finance: Evidence from a bibliometric review
Mwonge, L. A., Akyoo, A. M., & Malaki, C. O. (2026) · Scientific African · doi:10.1016/j.sciaf.2026.e03446
Factors behind high caesarean section in Bangladesh: insight from a systematic literature review and meta-synthesis
Ali, M., Younger, A., Votruba, N., & Hirst, J. E. (2026) · BMJ Public Health · doi:10.1136/bmjph-2025-003550 · Cited by 1
Pediatric nurses' perspectives of integrating spiritual care into everyday practice: An integrative review
Htay, Y. M. H., Foster, M., & Mowat, R. M. (2026) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2026.05.044
Application of Digital Twin Technology to Enhance Chronic Diseases Management: A Systematic Review
Zarei, A., Gholamzadeh, M., & Asadi, F. (2026) · International Journal of Telemedicine and Applications · doi:10.1155/ijta/2299762
Intervenciones de enfermería para prevenir infecciones asociadas a la Atención de Salud en Hospitales Latinoamericanos: protocolo de Revisión Sistemática de métodos mixtos con énfasis en Ecuador
Almeida, M. Y. G., Orbea, S. T. R., Veloz, C. M. P., & Gómez, W. C. G. (2026) · Revista Multidisciplinar Epistemología de las Ciencias · doi:10.71112/sdx5g689
Feasibility and preliminary effects of mhealth supportive care interventions in adults with myeloid neoplasms: a systematic review
Chen, K., Gou, C., Wu, Z., & Guo, L. (2026) · Supportive Care in Cancer · doi:10.1007/s00520-026-10887-5
Impact of Life‐Context Factors on Chronic Rhinosinusitis 22‐Item Sinonasal Outcomes Test: Systematic Review + Meta‐Analysis
Jade, K. M., Scangas, G. A., Lee, V. S., Man, L., McCoul, E. D., Guevara, G., Nourmahnad, A., Reynolds, J. M., & Levine, C. G. (2026) · Otolaryngology · doi:10.1002/ohn.70311
Challenges and opportunities in the implementation of active learning methods in occupational safety and health training
Rocha, M. C., Dores, A. R., Ferreira, D. F., & Rodrigues, M. A. (2026) · Work · doi:10.1177/10519815261456687
No difference in residual laxity following all types of arthroscopic treatment of chronic ankle instability: A meta-analysis
Giunta, J.-C., Agu, C., & van Rooij, F. (2026) · Foot and Ankle Surgery · doi:10.1016/j.fas.2026.06.007
Construction of a faculty competency model for medical simulation education integrated with GenAI: a multi-method quantitative study based on the perspective of medical students
Zhu-ge, Y., Yao, X.-T., Mei, H., Yao, H.-X., & Chen, Q. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09734-2
“A multi-method approach was employed incorporating the Delphi technique [5] and a cross-sectional survey for empirical validation.”
Screening tools and programmes for intimate partner violence in antenatal and postpartum healthcare across Europe: a scoping review with quality assessment
Normann, A. K., Paradiso, M. N., Brülle, A., Garzon, S., Andreasen, K., Donisi, V., Piccolo, L. D., Manfredi, P., Jørgensen, J. S., & Linde, D. S. (2026) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-026-09473-8
“Due to the different design of the studies, several quality assessment tools were employed: The Appraisal tool for Cross-Sectional Studies (AXIS) [17] was applied for cross-sectional studies [7, 9, 18–27] and for mixed-method studies [28–31] the Mixed-Methods Appraisal Tool (MMAT) [32] was used.”
Efficacy of Virtual Reality–Based Mindfulness Interventions: Systematic Review and Meta-Analysis
Barker, S., Miles‐Novelo, A., Rizzo, A., & Tuma, R. M. (2026) · JMIR XR and spatial computing. · doi:10.2196/90003
“Although DerSimonian-Laird estimation is standard in small-sample meta-analyses, it may underestimate uncertainty under conditions of extreme heterogeneity; results should therefore be interpreted conservatively [14].”
Influential

Authors: Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
Affiliations: McGill University (CA)
Universitat Oberta de Catalunya (ES)
University of Warwick (GB)
University of Canberra (AU)
Université de Sherbrooke (CA)
Université Laval (CA)
University of Sheffield (GB)
Institut National de la Recherche Scientifique (CA)
Corresponding author: Pluye, Pierre
ORCID: Hong, Quan Nha
Pluye, Pierre
Fàbregues, Sergi
Bartlett, Gillian
Boardman, Felicity
Cargo, Margaret
Dagenais, Pierre
Gagnon, Marie-Pierre
Griffiths, Frances
Nicolau, Belinda
O’Cathain, Alicia
Rousseau, Marie-Claude
Vedel, Isabelle
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2019-07
Volume: 111
Pages: 49-59.e1
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2019.03.008
OpenAlex: W2924322406
Cited by: 1,082
References: 50
Citations by year: 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
FWCI: 168.24
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by-nc-nd
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
Locations: Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
PubMed (publishedVersion)
The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
HAL (submittedVersion, cc-by-nc-nd)
eScholarship@McGill (McGill) (submittedVersion)
EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
Warwick Research Archive Portal (University of Warwick) (submittedVersion)
Topics: Social Sciences › Sociology and Political Science › Delphi Technique in Research
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Health Sciences › Nursing › Research and Theory › Health Education and Validation
Keywords: Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
MeSH terms: Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks