Cited by 1,082Delphi Technique in ResearchMeta-analysis and systematic reviewsHealth Education and Validation

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

Open access

Hong, Quan Nha, Pluye, Pierre, Fàbregues, Sergi, Bartlett, Gillian, Boardman, Felicity, Cargo, Margaret, Dagenais, Pierre, Gagnon, Marie-Pierre, Griffiths, Frances, Nicolau, Belinda, O’Cathain, Alicia, Rousseau, Marie-Claude, Vedel, Isabelle·2019·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2019.03.008

Open this paper in CoChat — free account Open PDF

Abstract

OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2019.03.008

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
Method
Achieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
Background
Determination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
Background
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
Method
Is the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
Background
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
Method
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657

Show all 50 references

Sorted by citations

Items 101–200 of 1,001

Effects of Physical Exercise Interventions on Stereotyped Motor Behaviours in Children with ASD: A Meta-Analysis
Teh, E. J., Vijayakumar, R., Tan, T. X. J., & Yap, M. J.-M. (2021) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-021-05152-z · Cited by 42
Frailty in People with Chronic Limb Threatening Ischemia and Diabetes-Related Foot Ulcers: A Systematic Review
Fernando, M. E., Blanchette, V., Mishra, R. K., Zulbaran‐Rojas, A., Rowe, V. L., Mills, J. L., Armstrong, D. G., & Najafi, B. (2022) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2022.09.057 · Cited by 37
Barriers and facilitators to deprescribing of cardiovascular medications: a systematic review
Brunner, L., Rodondi, N., & Aubert, C. E. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061686 · Cited by 37
Variations of mixed methods reviews approaches: A case study
Hong, Q. N., Rees, R. W., Sutcliffe, K., & Thomas, J. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1437 · Cited by 52
Self-citation
Flow in youth sport, physical activity, and physical education: A systematic review
Jackman, P. C., Dargue-Fox, E. J., Johnston, J., & Hawkins, R. M. (2020) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2020.101852 · Cited by 51
Development of the ASSESS tool: a comprehenSive tool to Support rEporting and critical appraiSal of qualitative, quantitative, and mixed methods implementation reSearch outcomes
Ryan, N., Vieira, D. L., Gyamfi, J., Ojo, T., Shelley, D., Ogedegbe, O., Iwelunmor, J., & Peprah, E. K. (2022) · Implementation Science Communications · doi:10.1186/s43058-021-00236-4 · Cited by 38
“To develop our tool, a list of new items and items from existing tools were combined, including elements from the TIDieR checklist [21], StaRI checklist [20], MMAT tool [23], implementation outcomes taxonomy [8], and FRAME framework [22].”
BackgroundMethod
Identifying stigmatizing language in clinical documentation: A scoping review of emerging literature
Barcelona, V., Scharp, D., Idnay, B., Moen, H., Cato, K., & Topaz, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303653 · Cited by 45
Peer support for people living with rare or young onset dementia: An integrative review
Sullivan, M. P., Williams, V., Grillo, A., McKee‐Jackson, R., Camic, P. M., Windle, G., Stott, J. C. H., Brotherhood, E., & Crutch, S. J. (2022) · Dementia · doi:10.1177/14713012221126368 · Cited by 35
Conceptualizing Family Caregivers’ Use of Community Support Services: A Scoping Review
Choi, H., Reblin, M., & Litzelman, K. (2023) · The Gerontologist · doi:10.1093/geront/gnad039 · Cited by 35
An integrative review on mothers' experiences of online breastfeeding peer support: Motivations, attributes and effects
Moon, H., & Woo, K. (2021) · Maternal and Child Nutrition · doi:10.1111/mcn.13200 · Cited by 40
Nurses’ perceptions of their role with respect to promoting physical activity in adult patients: a systematic review
van Hell‐Cromwijk, M., Metzelthin, S., Schoonhoven, L., Verstraten, C., Kroeze, W., & Ginkel, J. M. D. M. (2021) · Journal of Clinical Nursing · doi:10.1111/jocn.15747 · Cited by 35
Feasibility, acceptability, and effectiveness of web-based and mobile PTSD Coach: a systematic review and meta-analysis
Bröcker, E., Suliman, S., Olff, M., & Seedat, S. (2023) · European Journal of Psychotraumatology · doi:10.1080/20008066.2023.2209469 · Cited by 35
“The mixed methods appraisal tool (MMAT) was applied to all manuscripts (Hong et al., 2019).”
Method
Challenges of Shared Decision-making by Clinicians and Patients With Low-risk Differentiated Thyroid Cancer
Yang, W., Lee, Y. K., Lorgelly, P. K., Rogers, S. N., & Kim, D. H. (2023) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2023.0101 · Cited by 37
The association between road traffic noise and myocardial infarction: A systematic review and meta-analysis
Khosravipour, M., & Khanlari, P. (2020) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2020.139226 · Cited by 35
Colour schemes to reduce stress response in the hygiene area of a space station: A Delphi study
Jiang, A., Foing, B. H., Schlacht, I. L., Yao, X., Cheung, V., & Rhodes, P. A. (2021) · Applied Ergonomics · doi:10.1016/j.apergo.2021.103573 · Cited by 34
Use of Facebook, Instagram, and Twitter for recruiting healthy participants in nutrition-, physical activity–, or obesity-related studies: a systematic review
Ellington, M., Connelly, J., Clayton, P., Lorenzo, C. Y., Collazo-Velazquez, C., Trak‐Fellermeier, M. A., & Palacios, C. S. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab352 · Cited by 34
Strategies to reduce waiting times in outpatient rehabilitation services for adults with physical disabilities: A systematic literature review
Dupuis, F., Déry, J., de Oliveira, F. C. L., Pécora, A. T. L., Gagnon, R., Harding, K. E., Camden, C., Roy, J., Lettre, J., Hudon, A., et al. (2022) · Journal of Health Services Research & Policy · doi:10.1177/13558196211065707 · Cited by 34
A systematic review and behaviour change technique analysis of remotely delivered alcohol and/or substance misuse interventions for adults
Howlett, N., García-Iglesias, J., Bontoft, C., Breslin, G., Bartington, S., Freethy, I., Malillos, M. H., Jones, J., Lloyd, N., Marshall, T., et al. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2022.109597 · Cited by 37
Diagnosis and treatment of Wernicke's encephalopathy: A systematic literature review
Cantu-Weinstein, A., Branning, R., Alamir, M., Weleff, J., Do, M., Nero, N., & Anand, A. (2024) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2024.01.005 · Cited by 34
Barriers and facilitators to physical activity participation for child, adolescent, and young adult cancer survivors: a systematic review
Adamovich, T., Watson, R. E., Murdoch, S., Giovino, L., Kulkarni, S., Luchak, M., & Smith‐Turchyn, J. (2022) · Journal of Cancer Survivorship · doi:10.1007/s11764-022-01217-9 · Cited by 38
Methodological parameters for upper airway assessment by cone-beam computed tomography in adults with obstructive sleep apnea: a systematic review of the literature and meta-analysis
Gurgel, M., Chaves, C. M., Cevidanes, L., Silva, P. G. D. B., Carvalho, F. S. R., Kurita, L. M., Cunha, T. C. A., Fabbro, C. D., & Costa, F. W. G. (2022) · Sleep And Breathing · doi:10.1007/s11325-022-02582-6 · Cited by 34
Sexual and reproductive health (SRH) needs for forcibly displaced adolescent girls and young women (10–24 years old) in humanitarian settings: a mixed-methods systematic review
Soeiro, R. E., Guida, J. P. D. S., da‐Costa‐Santos, J., & Costa, M. L. (2023) · Reproductive Health · doi:10.1186/s12978-023-01715-8 · Cited by 32
A decade of designing and implementing electronic health records in Sub-Saharan Africa: a scoping review
Mugauri, H. D., Chimsimbe, M., Shambira, G., Shamhu, S., Nyamasve, J., Munyanyi, M., Gongora, R., Zizhou, S., Gabida, M., & Makurumidze, R. (2025) · Global Health Action · doi:10.1080/16549716.2025.2492913 · Cited by 32
“The quality of the 30 included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT), appropriate for evaluating empirical studies of different designs (qualitative, quantitative, or mixed methods) [12,13].”
Knowledge, perceptions and experiences of nurses in antimicrobial optimization or stewardship in the intensive care unit
Padigos, J., Reid, S. A., Kirby, E. R., & Broom, J. K. (2020) · Journal of Hospital Infection · doi:10.1016/j.jhin.2020.12.003 · Cited by 33
Psychological factors associated with foot and ankle pain: a mixed methods systematic review
Cotchett, M., Frescos, N., Whittaker, G. A., & Bonanno, D. R. (2022) · Journal of Foot and Ankle Research · doi:10.1186/s13047-021-00506-3 · Cited by 32
Factors that contribute to turnover and retention amongst emergency department nurses: a scoping review
McIntyre, N., Crilly, J., & Elder, E. (2024) · International Emergency Nursing · doi:10.1016/j.ienj.2024.101437 · Cited by 31
A global overview of midwives’ working conditions: A rapid review of literature on positive practice environment
Carvajal, B., Hancock, A., Lewney, K., Hagan, K., Jamieson, S., & Cooke, A. (2023) · Women and Birth · doi:10.1016/j.wombi.2023.08.007 · Cited by 34
K-12 students' higher-order thinking skills: Conceptualization, components, and evaluation indicators
Liu, J., Liu, Z., Wang, C., Xu, Y., Chen, J., & Cheng, Y. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101551 · Cited by 65
Barriers to physical activity in women from ethnic minority groups: a systematic review
Batalha, A. P. D. B., Marçal, I. R., Main, E., & Ghisi, G. L. D. M. (2025) · BMC Women s Health · doi:10.1186/s12905-025-03877-y · Cited by 31
“…and culture; in the Southeast Asia, culture, family responsibilities, and environment; and in the Western Pacific, culture, physical conditions, [ 15 ] † T h e t o t a l n u m b e r o f b a rr i e r s r e p r e s e n t s t h e s u m o f b a rr i e r s r e p o r t e d b y w o m e n f r o m e t hn…”
The current significance of the FOUR score: A systematic review and critical analysis of the literature
Anestis, D. M., Tsitsopoulos, P. P., Tsonidis, C. A., & Foroglou, N. (2019) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2019.116600 · Cited by 31
Students’ perceptions on the recognition of holistic competency achievement: A systematic mixed studies review
Chan, C. K. Y., & Chen, S. W. (2021) · Educational Research Review · doi:10.1016/j.edurev.2021.100431 · Cited by 31
A systematic review on the impact of pharmacist‐provided services on patients’ health outcomes in Arab countries
El‐Awaisi, A., Al‐Shaibi, S., Al‐Ansari, R., Naseralallah, L. M. A., & Awaisu, A. (2022) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/jcpt.13633 · Cited by 31
Job satisfaction of advanced practice nurses in cancer care: A systematic review
Geese, F., Zwakhalen, S., Lucien, B., & Hahn, S. (2022) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2021.102089 · Cited by 30
Measurement and assessment of fidelity and competence in nonspecialist-delivered, evidence-based behavioral and mental health interventions: A systematic review
Bond, L., Simmons, E. C., & Sabbath, E. L. (2022) · SSM - Population Health · doi:10.1016/j.ssmph.2022.101249 · Cited by 33
“The tool has been recently revised and includes criteria on five categories of studies, including mixed methods (Hong et al., 2019).”
Method
Factors influencing the reporting of medication errors and near misses among nurses: A systematic mixed methods review
Braiki, R., Douville, F., & Gagnon, M. (2024) · International Journal of Nursing Practice · doi:10.1111/ijn.13299 · Cited by 37
Self-citation
Impact of retirement transition on health, well-being and health behaviours: critical insights from an overview of reviews
Vigezzi, G. P., Barbati, C., Maggioni, E., Stenholm, S., Odone, A., Amerio, A., Ardito, C., Bertuccio, P., Costa, G., d’Errico, A., et al. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.118049 · Cited by 36
“Quality appraisal of included studies (as reported in Table 3) was performed following the mixed-methods appraisal tool (MMAT) (Hong et al., 2019) in Baer (Baer et al., 2020), the Critical Appraisal Skills Programme (CASP) checklist (https://casp-uk.net/casp-tools-check-lists/) in Barnett (Barnett…”
A mixed-methods systematic review of suicide prevention interventions involving multisectoral collaborations
Pearce, T., Maple, M. J., Wayland, S., McKay, K., Woodward, A., Brooks, A., & Shakeshaft, A. P. (2022) · Health Research Policy and Systems · doi:10.1186/s12961-022-00835-0 · Cited by 32
The Effectiveness and Feasibility of Psychological Interventions for Populations Under Ongoing Threat: A Systematic Review
Yim, S. H., Lorenz, H., & Salkovskis, P. M. (2023) · Trauma Violence & Abuse · doi:10.1177/15248380231156198 · Cited by 34
“The items were designed and revised through a Delphi study (Hong et al., 2019) to increase content validity.”
Method
Navigating the AI Revolution: Challenges and Opportunities for Integrating Emerging Technologies into Knowledge Management Systems. Systematic Literature Review
Gelashvili-Luik, T., Vihma, P., & Pappel, I. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1595930 · Cited by 30
“Studies were classified as high (scores of 12–14), medium (8–11), or low quality (0–7), using thresholds informed by CASP-based scoring approaches and the Mixed Methods Appraisal Tool (MMAT) framework (Hong et al., 2019).”
Barriers and enablers in the implementation and sustainability of toothbrushing programs in early childhood settings and primary schools: a systematic review
Chandio, N., Micheal, S., Tadakmadla, S. K., Sohn, W., Cartwright, S., White, R., Sanagavarapu, P., Parmar, J. S., & Arora, A. (2022) · BMC Oral Health · doi:10.1186/s12903-022-02270-7 · Cited by 29
The impact of stigma on the management of type 1 diabetes: A systematic review
Embick, R., Jackson, M. C., & Stewart, R. (2024) · Diabetic Medicine · doi:10.1111/dme.15299 · Cited by 29
“35 Instead, it is recommended to utilise the MMAT as a tool to understand what areas to report on.”
Method
The Intersection Between Food Literacy and Sustainability: A Systematic Quantitative Literature Review
McManus, S., Pendergast, D., & Kanasa, H. (2025) · Sustainability · doi:10.3390/su17020459 · Cited by 29
Cultural adaptations of obesity‐related behavioral prevention interventions in early childhood: A systematic review
Marshall, S. K., Taki, S., Laird, Y., Love, P., Wen, L. M., & Rissel, C. E. (2021) · Obesity Reviews · doi:10.1111/obr.13402 · Cited by 33
“The MMAT is a reliable and valid tool for assessing the methodological quality.(40,41) MMAT focuses on core items representing the overall quality of evidence and risk of bias, which may impact the validity of the study findings.”
Method
The changes in the nursing practice environment brought by COVID-19 and improvement recommendations from the nurses’ perspective: a cross-sectional study
Cheng, J., Zhu, L., Zuo, Q., Weixue, P., & Jiang, X. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08135-7 · Cited by 29
Factors influencing symptom appraisal and help-seeking of older adults with possible cancer: a mixed-methods systematic review
Jones, D. J., di Martino, E., Bradley, S. H., Essang, B., Hemphill, S., Wright, J. M., Renzi, C., Surr, C., Clegg, A. P., & Neal, R. D. (2022) · British Journal of General Practice · doi:10.3399/bjgp.2021.0655 · Cited by 29
The effects of interpersonal development programmes with sport coaches and parents on youth athlete outcomes: A systematic review and meta-analysis
Bengtsson, D., Stenling, A., Nygren, J., Ntoumanis, N., & Ivarsson, A. (2023) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2023.102558 · Cited by 42
A scoping review of rebel nurse leadership: Descriptions, competences and stimulating/hindering factors
de Kok, E., Weggelaar‐Jansen, A. M., Schoonhoven, L., & Lalleman, P. (2021) · Journal of Clinical Nursing · doi:10.1111/jocn.15765 · Cited by 28
Olfactory stimulation for people with dementia: A rapid review
D’Andrea, F., Tischler, V., Dening, T., & Churchill, A. (2022) · Dementia · doi:10.1177/14713012221082377 · Cited by 28
Influencing factors of novice pilot SA based on DEMATEL-AISM method: From pilots’ view
Chen, H., Liu, S., Wanyan, X., Pang, L., Dang, Y., Zhu, K., & Yu, X. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e13425 · Cited by 28
Contents and effectiveness of patient‐ and family‐centred care interventions in adult intensive care units: A systematic review
Joo, Y., Jang, Y., & Kwon, O. Y. (2024) · Nursing in Critical Care · doi:10.1111/nicc.13105 · Cited by 28
Patient-reported experiences and outcomes of virtual care during COVID-19: a systematic review
Bajgain, B., Rabi, S., Ahmed, S., Kiryanova, V., Fairie, P., & Santana, M. J. (2023) · Journal of Patient-Reported Outcomes · doi:10.1186/s41687-023-00659-8 · Cited by 26
Dementia and Mental Health During the COVID-19 Pandemic: A Systematic Review
Gaigher, J. M., Lacerda, I. B., & Dourado, M. C. N. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.879598 · Cited by 29
Inequities in neurological care: Access to services, care gaps, and other barriers—A systematic review
Fornari, A., Lanza, M., Guastafierro, E., Marcassoli, A., Sismondo, P., Curatoli, C., Raggi, A., & Leonardi, M. (2024) · European Journal of Neurology · doi:10.1111/ene.16553 · Cited by 32
“However, MMAT is the only critical appraisal that includes specific criteria for mixed methods studies using a combination of individual component and mixed methods approaches that is distinct from the other tools [ 38 ].”
A Scoping Review of Health Outcomes Among Transgender Migrants
Castro, V. A., King, W. M., Augustaitis, L., Saylor, K., & Gamarel, K. E. (2021) · Transgender Health · doi:10.1089/trgh.2021.0011 · Cited by 28
Effects of age simulation suits on psychological and physical outcomes: a systematic review
Gerhardy, T. H., Schlomann, A., Wahl, H., & Schmidt, L. I. (2022) · European Journal of Ageing · doi:10.1007/s10433-022-00722-1 · Cited by 26
Stimulating implementation of clinical practice guidelines in hospital care from a central guideline organization perspective: A systematic review
Thoonsen, A. C., Schoten, S. M. V., Merten, H., van Beusekom, I., Schoonmade, L. J., Delnoij, D. M. J., & de Bruijne, M. C. (2024) · Health Policy · doi:10.1016/j.healthpol.2024.105135 · Cited by 31
Podcasts as a tool for promoting health-related behaviours: A scoping review
Robins, B., Delaney, T. M., Maher, C. A., & Singh, B. (2024) · Digital Health · doi:10.1177/20552076241288630 · Cited by 30
Digital health interventions to improve eating behaviour of people with a lower socioeconomic position: a scoping review of behaviour change techniques
Ronteltap, A., Bukman, A. J., Nagelhout, G. E., Hermans, R. C. J., Hosper, K., Haveman‐Nies, A., Lupker, R., & Bolman, C. A. W. (2022) · BMC Nutrition · doi:10.1186/s40795-022-00635-3 · Cited by 28
The “Face” of Coach Development: A Systematic Review of the Role of the Coach Developer
Jones, T., Allen, J. B., & MacDonald, S. C. (2023) · International Sport Coaching Journal · doi:10.1123/iscj.2022-0017 · Cited by 26
Factors Influencing the Salience of Military/Veteran Identity Post Discharge: A Scoping Review
Dolan, G., McCauley, M., & Murphy, D. (2022) · Journal of Veterans Studies · doi:10.21061/jvs.v8i1.333 · Cited by 25
The role of pets in the support systems of community-dwelling older adults: a qualitative systematic review
Reniers, P. W. A., Declercq, I., Hediger, K., Enders‐Slegers, M., Gerritsen, D. L., & Leontjevas, R. (2022) · Aging & Mental Health · doi:10.1080/13607863.2022.2141196 · Cited by 37
Animal-assisted and robotic animal-assisted interventions within dementia care: A systematic review
Shoesmith, E., Surr, C., & Ratschen, E. (2023) · Dementia · doi:10.1177/14713012231155985 · Cited by 25
“The tool’s validity and reliability have been established (Hong et al., 2019; Pace et al., 2012).”
Actions targeting the integration of peer workforces in mental health organisations: a mixed-methods systematic review
Reeves, V., McIntyre, H., Loughhead, M., Halpin, M. A., & Procter, N. G. (2024) · BMC Psychiatry · doi:10.1186/s12888-024-05664-9 · Cited by 25
A rapid review of stocking and marketing practices used to sell sugar‐sweetened beverages in U.S. food stores
Houghtaling, B., Holston, D., Szocs, C., Penn, J., Qi, D., & Hedrick, V. E. (2020) · Obesity Reviews · doi:10.1111/obr.13179 · Cited by 26
Barriers and Facilitators Influencing Arab Muslim Immigrant and Refugee Women’s Breast Cancer Screening: A Narrative Review
Racine, L., & Andsoy, İ. I. (2022) · Journal of Transcultural Nursing · doi:10.1177/10436596221085301 · Cited by 25
Family Involvement in Pain Management for People Living With Dementia: An Integrative Review
Pu, L., Chen, H., Jones, C., & Moyle, W. (2022) · Journal of Family Nursing · doi:10.1177/10748407221114502 · Cited by 25
“The MMAT is a valid and reliable critical appraisal tool designed for systematic mixed methods studies (Hong et al., 2019).”
Method
The contribution of primary care practitioners to interventions reducing loneliness and social isolation in older people—An integrative review
Thompson, C., Halcomb, E., & Masso, M. R. (2023) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.13151 · Cited by 28
Exploring mobile mixed reality for critical thinking in nursing and healthcare education: A systematic review
Stretton, T., Cochrane, T., Sevigny, C. P., & Rathner, J. A. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.106072 · Cited by 32
Developing a quality framework for community pharmacy: a systematic review of international literature
Hindi, A. M. K., Campbell, S., Jacobs, S., & Schafheutle, E. I. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-079820 · Cited by 24
Practice recommendations for culturally sensitive communication at the end of life in intensive care: A modified eDelphi study
Brooks, L. A., Manias, E., Rasmussen, B., & Bloomer, M. J. (2024) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2024.103814 · Cited by 25
A systematic review of research into coach perspectives and behaviours regarding doping and anti-doping
Barnes, L. T., Patterson, L. B., & Backhouse, S. H. (2020) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2020.101780 · Cited by 31
Impact of virtual reality interventions on haemodialysis patients: A scoping review
Omonaiye, O., Smyth, W., & Nagle, C. (2021) · Journal of Renal Care · doi:10.1111/jorc.12362 · Cited by 25
Being active with a total hip or knee prosthesis: a systematic review into physical activity and sports recommendations and interventions to improve physical activity behavior
Mooiweer, Y., Stevens, M. J., van den Akker‐Scheek, I., PAIR study group, Barone, G., Benvenuti, F., Berteanu, M., Bragonzoni, L., Ciobanu, I., Dallari, D., et al. (2022) · European Review of Aging and Physical Activity · doi:10.1186/s11556-022-00285-1 · Cited by 24
Barriers and facilitators to mobility of patients hospitalised on an acute medical ward: a systematic review
Mani, H., Möri, C., Mattmann, M., Liechti, F. D., Inauen, J., Aujesky, D., Donzé, J. D., & Aubert, C. E. (2022) · Age and Ageing · doi:10.1093/ageing/afac159 · Cited by 24
Restored quality of life-based approach (REQUALIFE) for urban seismic resilience assessment: Quantitative method
Pan, S., Zhao, Z., Lim, H. W., Li, N., & Fang, D. (2022) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2022.103169 · Cited by 25
Prevalence, Predictors, and Experience of Moral Suffering in Nursing and Care Home Staff during the COVID-19 Pandemic: A Mixed-Methods Systematic Review
Laher, Z., Robertson, N., Harrad-Hyde, F., & Jones, C. R. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159593 · Cited by 32
Spinal manipulation and mobilisation in the treatment of infants, children, and adolescents: a systematic scoping review
Milne, N., Longeri, L., Patel, A., Pool, J. J. M., Olson, K., Basson, A., & Gross, A. (2022) · BMC Pediatrics · doi:10.1186/s12887-022-03781-6 · Cited by 24
Healthcare professionals’ knowledge, confidence and attitudes in relation to psychosis care: A systematic review
Chen, J., El‐Den, S., Pham, L., O’Reilly, C. L., & Collins, J. C. (2023) · International Journal of Social Psychiatry · doi:10.1177/00207640231194490 · Cited by 24
The effectiveness of online interventions for patients with gynecological cancer: An integrative review
Lin, H., Ye, M., Chan, S. W., Zhu, J., & He, H. G. (2020) · Gynecologic Oncology · doi:10.1016/j.ygyno.2020.04.690 · Cited by 25
Barriers and enablers to medication adherence in glaucoma: A systematic review of modifiable factors using the Theoretical Domains Framework
Bott, D., Subramanian, A., Edgar, D. F., Lawrenson, J. G., & Campbell, P. (2023) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13245 · Cited by 23
Practical Guide to Achieve Rigor and Data Integration in Mixed Methods Research
Lorenzini, E., Galeano, S. P. O., de Antioquia, U., Schmidt, C. R., Cañón-Montañéz, W., & de Antioquia, U. (2024) · Investigación y Educación en Enfermería · doi:10.17533/udea.iee.v42n3e02 · Cited by 32
Effects of non-facilitated meaningful activities for people with dementia in long-term care facilities: A systematic review
Jones, C., Liu, F., Murfield, J., & Moyle, W. (2020) · Geriatric Nursing · doi:10.1016/j.gerinurse.2020.06.001 · Cited by 23
“The MMAT consists of a 7-question checklist and was chosen due to its applicability to critically appraise study designs that involve both randomised and non-randomised controlled trials, ease of use and established validity.(35) The level of agreement between the two authors was excellent (k = .”
Exploring the experiences of living with Lewy body dementia: An integrative review
Bentley, A., Morgan, T., Salifu, Y., & Walshe, C. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14932 · Cited by 23
“Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2019), and Gough's weight of evidence framework (Gough, 2007).”
Method
Effectiveness of art therapy for people with eating disorders: A mixed methods systematic review
Griffin, C., Fenner, P., Landorf, K. B., & Cotchett, M. (2021) · The Arts in Psychotherapy · doi:10.1016/j.aip.2021.101859 · Cited by 23
Outcomes of participation in parkrun, and factors influencing why and how often individuals participate: A systematic review of quantitative studies
Peterson, B. J., Withers, B., Hawke, F., Spink, M. J., Callister, R., & Chuter, V. (2022) · Journal of Sports Sciences · doi:10.1080/02640414.2022.2086522 · Cited by 23
“, 2019; Stevinson & Hickson, 2014) stated an acceptable survey response rate (60% or above; Hong et al., 2019), with one reporting a 1% response rate (A Grunseit et al.”
BackgroundMethodInfluential
Instruments to assess frailty in community dwelling older adults: A systematic review
Rasiah, J., Gruneir, A., Oelke, N. D., Estabrooks, C. A., Holroyd-Leduc, J. M., & Cummings, G. G. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104316 · Cited by 24
A systematic review of U.S. nursing faculty's knowledge, awareness, inclusion, and perceived importance of sexual and gender minority-related content in nursing curricula
Moore, S. E., Coleman, C. L., Hughes, T. L., Dorsen, C. G., Smith, S. K., Bower, K. M., Clochesy, J. M., Clark, K. D., & Sherman, A. D. F. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2023.101950 · Cited by 23
Systematic Review: Child Psychiatry Access Program Outcomes
Lee, C. M., Yonek, J. C., Lin, B., Bechelli, M., Steinbuchel, P. H., Fortuna, L. R., & Mangurian, C. V. (2023) · JAACAP Open · doi:10.1016/j.jaacop.2023.07.003 · Cited by 24
“,”
Physical activity referral scheme components: a study protocol for systematic review and meta-regression
Mino, E., Geidl, W., Naber, I., Weißenfels, A., Klamroth, S., Gelius, P. C., Abu‐Omar, K., & Pfeifer, K. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-049549 · Cited by 22
Factors influencing foot care behaviour among patients with diabetes: An integrative literature review
Woo, M. W. J., & Cui, J. (2023) · Nursing Open · doi:10.1002/nop2.1710 · Cited by 22
Evaluating the Adoption of mHealth Technologies by Community Health Workers to Improve the Use of Maternal Health Services in Sub-Saharan Africa: Systematic Review
Kachimanga, C., Zaniku, H. R., Divala, T. H., Ket, J. C. F., Mukherjee, J. S., Palazuelos, D., Kulinkina, A. V., Abejirinde, I. O., & van den Akker, T. (2024) · JMIR mhealth and uhealth · doi:10.2196/55819 · Cited by 37
Family Resilience and Its Influencing Factors in Patients With Cancer and Their Family Members: A Systematic Review
Qiu, X., Mao, J., Wang, C., Yang, X., & Li, Q. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17485 · Cited by 22
Primary health care roles of community pharmacists in low- and middle-income countries: a mixed methods systematic review
Hedima, E. W., & Okoro, R. N. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13387-0 · Cited by 23
“It has been employed in several systematic reviews [43, 52, 70, 124, 130, 131, 134, 143, 144] and has proven to be a reliable critical evaluation tool [68, 119, 151].”
Providing Medical Information to Older Adults in a Web-Based Environment: Systematic Review
McLean, B., Hossain, N., Donison, V., Gray, M., Durbano, S., Haase, K. R., Alibhai, S. M. H., & Puts, M. T. E. (2021) · JMIR Aging · doi:10.2196/24092 · Cited by 22
Effect of mental health collaborative care models on primary care provider outcomes: an integrative review
Holmes, A., & Chang, Y. (2022) · Family Practice · doi:10.1093/fampra/cmac026 · Cited by 22
Psychosocial interventions targeting parenting distress among parents with cancer – A systematic review and narrative synthesis of available interventions
Strandh, M. R., Hovén, E., Sörensdotter, R., Stålberg, K., Enebrink, P., Ljungman, L., & Wikman, A. (2023) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2023.104119 · Cited by 22
Gamification in eHealth for Chronic Disease Self-Management in Youth: A Systematic Review
Stutvoet, M. D., Levelt, L., Hrehovcsik, M. M., Veer, J. V., Visch, V., Bramer, W. M., Hillegers, M. H. J., Veltkamp, R. C., Nijhof, S. L., & Estévez‐López, F. (2024) · Games for Health Journal · doi:10.1089/g4h.2023.0111 · Cited by 26
Tuberculosis active case-finding interventions and approaches for prisoners in sub-Saharan Africa: a systematic scoping review
Kuupiel, D., Vezi, P. N., Bawontuo, V., Osei, E., & Mashamba-Thompson, T. P. (2020) · BMC Infectious Diseases · doi:10.1186/s12879-020-05283-1 · Cited by 22
A scoping review to map the concept, content, and outcome of wilderness programs for childhood cancer survivors
Jong, M., Lown, E. A., Schats, W., Mills, M. L., Otto, H. R., Gabrielsen, L. E., & Jong, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0243908 · Cited by 22
Candidate Factors Maintaining Social Anxiety in the Context of Psychotic Experiences: A Systematic Review
Aunjitsakul, W., McGuire, N., McLeod, H. J., & Gumley, A. I. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbab026 · Cited by 28

Authors: Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
Affiliations: McGill University (CA)
Universitat Oberta de Catalunya (ES)
University of Warwick (GB)
University of Canberra (AU)
Université de Sherbrooke (CA)
Université Laval (CA)
University of Sheffield (GB)
Institut National de la Recherche Scientifique (CA)
Corresponding author: Pluye, Pierre
ORCID: Hong, Quan Nha
Pluye, Pierre
Fàbregues, Sergi
Bartlett, Gillian
Boardman, Felicity
Cargo, Margaret
Dagenais, Pierre
Gagnon, Marie-Pierre
Griffiths, Frances
Nicolau, Belinda
O’Cathain, Alicia
Rousseau, Marie-Claude
Vedel, Isabelle
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2019-07
Volume: 111
Pages: 49-59.e1
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2019.03.008
OpenAlex: W2924322406
Cited by: 1,082
References: 50
Citations by year: 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
FWCI: 168.24
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by-nc-nd
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
Locations: Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
PubMed (publishedVersion)
The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
HAL (submittedVersion, cc-by-nc-nd)
eScholarship@McGill (McGill) (submittedVersion)
EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
Warwick Research Archive Portal (University of Warwick) (submittedVersion)
Topics: Social Sciences › Sociology and Political Science › Delphi Technique in Research
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Health Sciences › Nursing › Research and Theory › Health Education and Validation
Keywords: Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
MeSH terms: Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks