The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Open accessAbstract
OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
From The Mental Health of Elite-Level Coaches: A Systematic Scoping Review (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodAchieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 2×
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
BackgroundDetermination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2×
“78 or higher were indicative of good content validity [48].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
MethodIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1×
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
BackgroundPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
MethodSystematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,081
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 1×
“It is an essential step in systematic reviews to ensure that their recommendations and conclusions reflect the quality of the evidence reviewed [6].”
BackgroundIntegrating quantitative and qualitative research: How is it done
Bryman, A. (2006) · Qualitative Research · doi:10.1177/1468794106058877 · Cited by 3,354 · cited 1×
“We focused on surveys because they are often included in systematic mixed studies reviews, the existing tools have not been developed with experts, and surveys are among the most commonly used methods in mixed methods studies [42].”
MethodLikert scales: how to (ab)use them
Jamieson, S. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.02012.x · Cited by 2,423 · cited 1×
“Likert scales may have some limitations related to central tendency and desirability biases [50].”
BackgroundContent validity in psychological assessment: A functional approach to concepts and methods.
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995) · Psychological Assessment · doi:10.1037//1040-3590.7.3.238 · Cited by 2,387 · cited 2×
“The content validity of an assessment tool is defined as the degree to which criteria are relevant to and representative of their targeted construct [15].”
BackgroundEffectiveness and Efficiency: Random Reflections on Health Services
L'Etang, H. (1973) · Public Health · doi:10.1016/s0033-3506(73)80082-4 · Cited by 1,905
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
The Delphi Technique in Nursing and Health Research
Keeney, S., Hasson, F., & McKenna, H. (2011) · Wiley eBooks · doi:10.1002/9781444392029 · Cited by 1,568 · cited 5×
“Usually, a smaller sample, such as 10 to 15 participants, is considered sufficient for homogeneous samples [20].”
BackgroundMethodInfluentialThe Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis
Groves, R. M., & Peytcheva, E. (2008) · Public Opinion Quarterly · doi:10.1093/poq/nfn011 · Cited by 1,441 · cited 1×
“This concurs with studies reporting a weak association between response rate and nonresponse bias [28].”
ResultPace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 1×
“Previous studies on the interrater reliability of the MMAT reported that agreement scores ranged from poor to perfect [12,13].”
BackgroundSelf-citationAn Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 1×
“In addition, in interviews conducted with MMAT users to explore their views and experiences of the MMAT, concerns were raised about whether the tool included enough criteria to judge the quality of studies and criteria that were difficult to judge, in particular the criteria for qualitative and mixed methods studies […”
MethodSelf-citationLewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866 · cited 1×
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodDillman, D. A., Phelps, G., Tortora, R. D., Swift, K., Kohrell, J., Berck, J., & Messer, B. L. (2008) · Social Science Research · doi:10.1016/j.ssresearch.2008.03.007 · Cited by 851 · cited 1×
“Although no specific criteria on measurement and response rate reached high level of consensus, the research team decided not to exclude these two biases from the MMAT because they are often mentioned in the literature [27,40,41].”
BackgroundHong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754 · cited 2×
“The first was a review of systematic mixed studies reviews that was carried out in 2015 [22].”
BackgroundMethodSelf-citationReviewing studies with diverse designs: the development and evaluation of a new tool
Sirriyeh, R., Lawton, R., Gardner, P., & Armitage, G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01662.x · Cited by 715 · cited 1×
“Also, some tools such as the Quality Assessment Tool for Studies with Diverse Designs (QATSDD) [45] use a combination of generic and individual component approaches, with generic criteria applicable to several designs and specific criteria for qualitative and quantitative studies.”
MethodAppraising the quality of qualitative research
Walsh, D., & Downe, S. (2006) · Midwifery · doi:10.1016/j.midw.2005.05.004 · Cited by 523
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 454
How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques
Baker, J. A., Lovell, K., & Harris, N. (2006) · Nurse Researcher · doi:10.7748/nr2006.10.14.1.59.c6010 · Cited by 404 · cited 1×
“An expert is defined as an individual with knowledge and skills in a specific area [23].”
BackgroundSouto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Previous studies on the interrater reliability of the MMAT reported that agreement scores ranged from poor to perfect [12,13].”
BackgroundSelf-citationAppraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools
Majid, U., & Vanstone, M. (2018) · Qualitative Health Research · doi:10.1177/1049732318785358 · Cited by 301 · cited 1×
“For example, two recent literature reviews on tools for qualitative studies analyzed more than 100 tools [19,47].”
BackgroundA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 1×
“For example, Crowe and Sheppard [36] use a generic approach by proposing one set of criteria that could be applied to any design.”
BackgroundCarroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Compared to reporting quality criteria, methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results that are reported can be trustworthy [43].”
BackgroundCarroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235
Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research
Davern, M. J. (2013) · Health Services Research · doi:10.1111/1475-6773.12070 · Cited by 219 · cited 1×
“Although no specific criteria on measurement and response rate reached high level of consensus, the research team decided not to exclude these two biases from the MMAT because they are often mentioned in the literature [27,40,41].”
BackgroundAdjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 166 · cited 1×
“For the MMAT, we retained the criterion #6 (quantitative and qualitative components of the study are effectively integrated) because it was also mentioned in other studies as among the most prevalent criterion for assessing the quality of mixed methods studies [17,29].”
MethodWortman, P. M. (1994) · Cited by 154
Santiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.007 · Cited by 146 · cited 2×
“These terms are often used in critical appraisal tools of qualitative research [19].”
BackgroundA Conceptual Framework for Critical Appraisal in Systematic Mixed Studies Reviews
Hong, Q. N., & Pluye, P. (2019) · Journal of Mixed Methods Research · doi:10.1177/1558689818770058 · Cited by 139 · cited 1×
“A conceptual framework on the quality appraisal in systematic mixed studies reviews was developed in which three dimensions of quality were presented: reporting, conceptual, and methodological [16].”
BackgroundSelf-citationPluye, P., Bengoechea, E. G., Granikov, V., Kaur, N., & Tang, D. L. (2018) · International Journal of Multiple Research Approaches · doi:10.29034/ijmra.v10n1a3 · Cited by 128 · cited 1×
“[30] identified nine strategies for integrating phases, results, or data.”
BackgroundSelf-citationA proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121
Study Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells, K., & Littell, J. H. (2008) · Research on Social Work Practice · doi:10.1177/1049731508317278 · Cited by 99
Addressing quality in mixed methods research: a review and recommendations for a future agenda
Fàbregues, S., & Molina-Azorín, J. F. (2016) · Quality & Quantity · doi:10.1007/s11135-016-0449-4 · Cited by 99 · cited 2×
“In this latter review, the authors analyzed 64 articles on the quality of mixed methods studies and identified 46 criteria [25].”
BackgroundMethodSelf-citationPluye, P. (2013) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12017 · Cited by 71 · cited 1×
“According to our review, the MMAT is the only tool that includes specific criteria for mixed methods studies [44].”
MethodSelf-citationColle, F., Rannou, F., Revel, M., Fermanian, J., & Poiraudeau, S. (2002) · Archives of Physical Medicine and Rehabilitation · doi:10.1053/apmr.2002.35657 · Cited by 68
How to Appraise the Studies: An Introduction to Assessing Study Quality
Petticrew, M. P., & Roberts, H. M. (2006) · Wiley eBooks · doi:10.1002/9780470754887.ch5 · Cited by 65
(2012) · Cited by 52
Burls, A. (2015) · International Journal of Evidence-Based Practice for the Dental Hygienist · doi:10.11607/ebh.001516 · Cited by 43
Bunn, F., Trivedi, D. P., Alderson, P., Hamilton, L. K., Martin, A. E., Pinkney, E., & Iliffe, S. (2015) · Health Technology Assessment · doi:10.3310/hta19280 · Cited by 42 · cited 1×
“Systematic reviews are considered among the best available sources of research evidence and are increasingly relied on to inform decision-making [1].”
BackgroundFàbregues, S., Paré, M., & Meneses, J. (2018) · Journal of Mixed Methods Research · doi:10.1177/1558689817751774 · Cited by 36
Self-citation
Cited by
Sorted by citations
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2020) · JBI Evidence Synthesis · doi:10.11124/jbisrir-d-19-00169 · Cited by 747
“…to develop a bespoke tool for mixed methods primary studies or identify an already existing critical appraisal tool for use in JBI MMSRs. 24,25,35,36 Additionally, with regard to critical appraisal in the integrated approach, further investigation into how the appraisal results of…”
Harrison, R., Jones, B., Gardner, P. H., & Lawton, R. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06122-y · Cited by 570
“across a diverse range of study designs [99].”
Göttgens, I., & Oertelt‐Prigione, S. (2021) · JMIR mhealth and uhealth · doi:10.2196/28102 · Cited by 445
“At present, MMAT is the most comprehensive tool available for appraising multimethod studies [125].”
MethodCassar, S., Salmon, J., Timperio, A. F., Naylor, P., van Nassau, F., Ayala, A. M. C., & Koorts, H. (2019) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-019-0876-4 · Cited by 313
Screening and Testing for Hepatitis B Virus Infection: CDC Recommendations — United States, 2023
Conners, E. E., Panagiotakopoulos, L., Hofmeister, M. G., Spradling, P. R., Hagan, L. M., Harris, A. M., Rogers-Brown, J. S., Wester, C. W., Nelson, N. P., Contributors, et al. (2023) · MMWR Recommendations and Reports · doi:10.15585/mmwr.rr7201a1 · Cited by 310
“The Mixed Methods Appraisal Tool (MMAT) was selected because it is a validated tool for assessing nonrandomized analytic and descriptive studies, which comprise most of the HBV infection prevalence literature (29).”
Metadata
- Authors
- Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 49-59.e1
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2924322406
- Cited by
- 1,082
- References
- 50
- Citations by year
- 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
- FWCI
- 168.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
- MeSH terms
- Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
