Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Open accessAbstract
OBJECTIVE: The mixed methods appraisal tool (MMAT) was developed for critically appraising different study designs. This study aimed to improve the content validity of three of the five categories of studies in the MMAT by identifying relevant methodological criteria for appraising the quality of qualitative, survey, and mixed methods studies. STUDY DESIGN AND SETTING: First, we performed a literature review to identify critical appraisal tools and extract methodological criteria. Second, we conducted a two-round modified e-Delphi technique. We asked three method-specific panels of experts to rate the relevance of each criterion on a five-point Likert scale. RESULTS: A total of 383 criteria were extracted from 18 critical appraisal tools and a literature review on the quality of mixed methods studies, and 60 were retained. In the first and second rounds of the e-Delphi, 73 and 56 experts participated, respectively. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. These results led to modifications of eight of the 11 MMAT (version 2011) criteria. Specifically, we reformulated two criteria, replaced four, and removed two. Moreover, we added six new criteria. CONCLUSION: Results of this study led to improve the content validity of this tool, revise it, and propose a new version (MMAT version 2018).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A quality appraisal was performed using the Mixed Methods Appraisal Tool (MMAT).
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
From The Mental Health of Elite-Level Coaches: A Systematic Scoping Review (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681 · cited 1× · see citing works
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodDetermination and Quantification Of Content Validity
LYNN, M. R. (1986) · Nursing Research · doi:10.1097/00006199-198611000-00017 · Cited by 6,775 · cited 2× · see citing works
“78 or higher were indicative of good content validity [48].”
BackgroundResearch guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1× · see citing works
“The Delphi technique is used to reach consensus among a group of experts [20] and is particularly suitable to build consensus on issues that have limited or contradictory evidence [21].”
MethodAchieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 5,772 · cited 2× · see citing works
“Also, Fetters, Curry, and Creswell [31] identified three integration levels (design, methods, and interpretation/reporting).”
BackgroundIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273 · cited 1× · see citing works
“Polit, Beck, and Owen [49] recommended having 8 to 12 experts for the first round.”
BackgroundPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1× · see citing works
“Initial developments mainly focused on meta-analysis for addressing questions on the effectiveness of interventions, and the emphasis was on randomized controlled trials [2,3].”
MethodSystematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,074 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 1× · see citing works
“It is an essential step in systematic reviews to ensure that their recommendations and conclusions reflect the quality of the evidence reviewed [6].”
BackgroundIntegrating quantitative and qualitative research: How is it done
Bryman, A. (2006) · Qualitative Research · doi:10.1177/1468794106058877 · Cited by 3,354 · cited 1× · see citing works
“We focused on surveys because they are often included in systematic mixed studies reviews, the existing tools have not been developed with experts, and surveys are among the most commonly used methods in mixed methods studies [42].”
MethodLikert scales: how to (ab)use them
Jamieson, S. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.02012.x · Cited by 2,423 · cited 1× · see citing works
“Likert scales may have some limitations related to central tendency and desirability biases [50].”
BackgroundContent validity in psychological assessment: A functional approach to concepts and methods.
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995) · Psychological Assessment · doi:10.1037//1040-3590.7.3.238 · Cited by 2,387 · cited 2× · see citing works
“The content validity of an assessment tool is defined as the degree to which criteria are relevant to and representative of their targeted construct [15].”
BackgroundEffectiveness and Efficiency: Random Reflections on Health Services
L'Etang, H. (1973) · Public Health · doi:10.1016/s0033-3506(73)80082-4 · Cited by 1,925 · see citing works
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831 · see citing works
The Delphi Technique in Nursing and Health Research
Keeney, S., Hasson, F., & McKenna, H. (2011) · Wiley eBooks · doi:10.1002/9781444392029 · Cited by 1,568 · cited 5× · see citing works
“Usually, a smaller sample, such as 10 to 15 participants, is considered sufficient for homogeneous samples [20].”
BackgroundMethodInfluentialThe Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis
Groves, R. M., & Peytcheva, E. (2008) · Public Opinion Quarterly · doi:10.1093/poq/nfn011 · Cited by 1,441 · cited 1× · see citing works
“This concurs with studies reporting a weak association between response rate and nonresponse bias [28].”
ResultPace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 1× · see citing works
“Previous studies on the interrater reliability of the MMAT reported that agreement scores ranged from poor to perfect [12,13].”
BackgroundSelf-citationIntroduction to Systematic Reviews
(2013) · Active Learning Exercises · doi:10.21019/ale.2000.252 · Cited by 1,239 · see citing works
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 1× · see citing works
“In addition, in interviews conducted with MMAT users to explore their views and experiences of the MMAT, concerns were raised about whether the tool included enough criteria to judge the quality of studies and criteria that were difficult to judge, in particular the criteria for qualitative and mixed methods studies […”
MethodSelf-citationLewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001895 · Cited by 866 · cited 1× · see citing works
“Several approaches for rating the overall quality of evidence have been developed, such as Grading of Recommendations, Assessment, Development and Evaluations (GRADE) [38] and GRADE-Confidence in the Evidence from Reviews of Qualitative research (CERQual) [39].”
MethodDillman, D. A., Phelps, G., Tortora, R. D., Swift, K., Kohrell, J., Berck, J., & Messer, B. L. (2008) · Social Science Research · doi:10.1016/j.ssresearch.2008.03.007 · Cited by 851 · cited 1× · see citing works
“Although no specific criteria on measurement and response rate reached high level of consensus, the research team decided not to exclude these two biases from the MMAT because they are often mentioned in the literature [27,40,41].”
BackgroundHong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754 · cited 2× · see citing works
“The first was a review of systematic mixed studies reviews that was carried out in 2015 [22].”
BackgroundMethodSelf-citationAssessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Ansari, M., Berkman, N., Chang, S. M., Hartling, L., McPheeters, M., Santaguida, P., Shamliyan, T., Singh, K., Tsertsvadze, A., et al. (2012) · doi:10.23970/ahrqepcmethguide2 · Cited by 731 · see citing works
Reviewing studies with diverse designs: the development and evaluation of a new tool
Sirriyeh, R., Lawton, R., Gardner, P., & Armitage, G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01662.x · Cited by 715 · cited 1× · see citing works
“Also, some tools such as the Quality Assessment Tool for Studies with Diverse Designs (QATSDD) [45] use a combination of generic and individual component approaches, with generic criteria applicable to several designs and specific criteria for qualitative and quantitative studies.”
MethodAppraising the quality of qualitative research
Walsh, D., & Downe, S. (2006) · Midwifery · doi:10.1016/j.midw.2005.05.004 · Cited by 523 · see citing works
How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques
Baker, J. A., Lovell, K., & Harris, N. (2006) · Nurse Researcher · doi:10.7748/nr2006.10.14.1.59.c6010 · Cited by 404 · cited 1× · see citing works
“An expert is defined as an individual with knowledge and skills in a specific area [23].”
BackgroundSouto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1× · see citing works
“Previous studies on the interrater reliability of the MMAT reported that agreement scores ranged from poor to perfect [12,13].”
BackgroundSelf-citationAppraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools
Majid, U., & Vanstone, M. (2018) · Qualitative Health Research · doi:10.1177/1049732318785358 · Cited by 301 · cited 1× · see citing works
“For example, two recent literature reviews on tools for qualitative studies analyzed more than 100 tools [19,47].”
BackgroundA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 1× · see citing works
“For example, Crowe and Sheppard [36] use a generic approach by proposing one set of criteria that could be applied to any design.”
BackgroundCarroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1× · see citing works
“Compared to reporting quality criteria, methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results that are reported can be trustworthy [43].”
BackgroundCarroll, C., & Booth, A. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1128 · Cited by 235 · see citing works
Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research
Davern, M. J. (2013) · Health Services Research · doi:10.1111/1475-6773.12070 · Cited by 219 · cited 1× · see citing works
“Although no specific criteria on measurement and response rate reached high level of consensus, the research team decided not to exclude these two biases from the MMAT because they are often mentioned in the literature [27,40,41].”
BackgroundAdjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · see citing works
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 166 · cited 1× · see citing works
“For the MMAT, we retained the criterion #6 (quantitative and qualitative components of the study are effectively integrated) because it was also mentioned in other studies as among the most prevalent criterion for assessing the quality of mixed methods studies [17,29].”
MethodWortman, P. M. (1994) · Cited by 154 · see citing works
Santiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.007 · Cited by 146 · cited 2× · see citing works
“These terms are often used in critical appraisal tools of qualitative research [19].”
BackgroundA Conceptual Framework for Critical Appraisal in Systematic Mixed Studies Reviews
Hong, Q. N., & Pluye, P. (2019) · Journal of Mixed Methods Research · doi:10.1177/1558689818770058 · Cited by 139 · cited 1× · see citing works
“A conceptual framework on the quality appraisal in systematic mixed studies reviews was developed in which three dimensions of quality were presented: reporting, conceptual, and methodological [16].”
BackgroundSelf-citationPluye, P., Bengoechea, E. G., Granikov, V., Kaur, N., & Tang, D. L. (2018) · International Journal of Multiple Research Approaches · doi:10.29034/ijmra.v10n1a3 · Cited by 128 · cited 1× · see citing works
“[30] identified nine strategies for integrating phases, results, or data.”
BackgroundSelf-citationA proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121 · see citing works
Addressing quality in mixed methods research: a review and recommendations for a future agenda
Fàbregues, S., & Molina-Azorín, J. F. (2016) · Quality & Quantity · doi:10.1007/s11135-016-0449-4 · Cited by 99 · cited 2× · see citing works
“In this latter review, the authors analyzed 64 articles on the quality of mixed methods studies and identified 46 criteria [25].”
BackgroundMethodSelf-citationStudy Quality Assessment in Systematic Reviews of Research on Intervention Effects
Wells, K., & Littell, J. H. (2008) · Research on Social Work Practice · doi:10.1177/1049731508317278 · Cited by 95 · see citing works
Pluye, P. (2013) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12017 · Cited by 71 · cited 1× · see citing works
“According to our review, the MMAT is the only tool that includes specific criteria for mixed methods studies [44].”
MethodSelf-citationColle, F., Rannou, F., Revel, M., Fermanian, J., & Poiraudeau, S. (2002) · Archives of Physical Medicine and Rehabilitation · doi:10.1053/apmr.2002.35657 · Cited by 68 · see citing works
How to Appraise the Studies: An Introduction to Assessing Study Quality
Petticrew, M. P., & Roberts, H. M. (2006) · Wiley eBooks · doi:10.1002/9780470754887.ch5 · Cited by 65 · see citing works
(2012) · Cited by 52 · see citing works
Burls, A. (2015) · International Journal of Evidence-Based Practice for the Dental Hygienist · doi:10.11607/ebh.001516 · Cited by 43 · see citing works
Bunn, F., Trivedi, D. P., Alderson, P., Hamilton, L. K., Martin, A. E., Pinkney, E., & Iliffe, S. (2015) · Health Technology Assessment · doi:10.3310/hta19280 · Cited by 42 · cited 1× · see citing works
“Systematic reviews are considered among the best available sources of research evidence and are increasingly relied on to inform decision-making [1].”
BackgroundFàbregues, S., Paré, M., & Meneses, J. (2018) · Journal of Mixed Methods Research · doi:10.1177/1558689817751774 · Cited by 36 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [27][27] [Federal Committee on Statistical Methodology. Measuring and reporting sources of error in surveys. Washington DC: Statistical Policy Office, Office of Information and Regulatory Affairs, Office of Management and Budget; 2001.](http://refhub.elsevier.com/S0895-4356(18)30082-9/sref27) -
- [35][35] [Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions. Chichester, UK: Wiley Online Library; 2008.](http://refhub.elsevier.com/S0895-4356(18)30082-9/sref35) -
Cited by
Sorted by citations
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2020) · JBI Evidence Synthesis · doi:10.11124/jbisrir-d-19-00169 · Cited by 747 · see citing works
“…to develop a bespoke tool for mixed methods primary studies or identify an already existing critical appraisal tool for use in JBI MMSRs. 24,25,35,36 Additionally, with regard to critical appraisal in the integrated approach, further investigation into how the appraisal results of…”
Harrison, R., Jones, B., Gardner, P. H., & Lawton, R. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06122-y · Cited by 513 · see citing works
“across a diverse range of study designs [99].”
Göttgens, I., & Oertelt‐Prigione, S. (2021) · JMIR mhealth and uhealth · doi:10.2196/28102 · Cited by 388 · see citing works
“At present, MMAT is the most comprehensive tool available for appraising multimethod studies [125].”
MethodCassar, S., Salmon, J., Timperio, A. F., Naylor, P., van Nassau, F., Ayala, A. M. C., & Koorts, H. (2019) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-019-0876-4 · Cited by 312 · see citing works
Screening and Testing for Hepatitis B Virus Infection: CDC Recommendations — United States, 2023
Conners, E. E., Panagiotakopoulos, L., Hofmeister, M. G., Spradling, P. R., Hagan, L. M., Harris, A. M., Rogers-Brown, J. S., Wester, C. W., Nelson, N. P., Contributors, et al. (2023) · MMWR Recommendations and Reports · doi:10.15585/mmwr.rr7201a1 · Cited by 277 · see citing works
“The Mixed Methods Appraisal Tool (MMAT) was selected because it is a validated tool for assessing nonrandomized analytic and descriptive studies, which comprise most of the HBV infection prevalence literature (29).”
Whitehead, L., Talevski, J., Fatehi, F., & Beauchamp, A. (2023) · Journal of Medical Internet Research · doi:10.2196/42719 · Cited by 274 · see citing works
“Quality appraisal was undertaken using the 2018 version of the Mixed Methods Appraisal Tool developed for use in systematic reviews of mixed methods studies [29].”
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2021) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000282 · Cited by 214 · see citing works
Kaboré, S. S., Ngangue, P., Soubeiga, D., Barro, A., Pilabré, A. H., Bationo, N., Pafadnam, Y., Drabo, K. M., Hien, H., & Savadogo, L. G. B. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1014375 · Cited by 223 · see citing works
“It allows the appraisal of five methodological quality categories: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies (13, 14, 16, 17).”
BackgroundInfluentialSocial media’s influence on adolescents′ food choices: A mixed studies systematic literature review
Kucharczuk, A. J., Oliver, T. L., & Dowdell, E. B. (2021) · Appetite · doi:10.1016/j.appet.2021.105765 · Cited by 203 · see citing works
Chabbert, M., Panagiotou, D., & Wendland, J. (2021) · Journal of Reproductive and Infant Psychology · doi:10.1080/02646838.2020.1748582 · Cited by 161 · see citing works
“The latest version of this tool was used for this review (Hong et al., 2019).”
Vo, V., Chen, G., Aquino, Y. S. J., Carter, S. M., Nga, Q., & Woode, M. E. (2023) · Social Science & Medicine · doi:10.1016/j.socscimed.2023.116357 · Cited by 172 · see citing works
Krzeczkowska, A., Spalding, D. M., McGeown, W. J., Gow, A. J., Carlson, M. C., & Brown, L. A. (2021) · Ageing Research Reviews · doi:10.1016/j.arr.2021.101400 · Cited by 166 · see citing works
Wong, E., Mavondo, F., & Fisher, J. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-05383-3 · Cited by 146 · see citing works
“The assessment is made against ve criteria, scored as ‘Yes,’ ‘No’ or ‘Can’t tell’, and it was developed systematically (31).”
Palliative care education and its effectiveness: a systematic review
Li, W., Chhabra, J., & Singh, S. (2021) · Public Health · doi:10.1016/j.puhe.2021.02.033 · Cited by 129 · see citing works
Xu, J. H., Lio, A., Dhaliwal, H., Andrei, S., Balakrishnan, S., Nagani, U., & Samadder, S. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.06.070 · Cited by 151 · see citing works
Wilson, S. A., Byrne, P., Rodgers, S. E., & Maden, M. (2022) · Innovation in aging · doi:10.1093/geroni/igac002 · Cited by 123 · see citing works
Pimentel, J., Ansari, U., Omer, K., Gidado, Y., Baba, M. C., Andersson, N., & Cockcroft, A. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-2852-z · Cited by 124 · see citing works
Chauke, G. D., Nakwafila, O., Chibi, B., Sartorius, B. K. D., & Mashamba-Thompson, T. P. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e09716 · Cited by 129 · see citing works
Almohanna, A. A. S., Win, K. T., & Meedya, S. (2020) · Journal of Medical Internet Research · doi:10.2196/17361 · Cited by 122 · see citing works
Sutter, E. D., Zaçe, D., Boccia, S., Pietro, M. L. D., Geerts, D., Borry, P., & Huys, I. (2020) · Journal of Medical Internet Research · doi:10.2196/19129 · Cited by 112 · see citing works
“MMAT and CASP were used to criticize study aspects such as the aim and the methodology of the study [20,21].”
MethodKhosravipour, M., Khanlari, P., Khazaie, S., Khosravipour, H., & Khazaie, H. (2021) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101427 · Cited by 103 · see citing works
Yopa, D. S., Massom, D. M., Kiki, G. M., Sophie, R. W., Sylvie, F., Thiam, O., Zinaba, L., & Ngangue, P. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1252428 · Cited by 95 · see citing works
Niazkhani, Z., Toni, E., Cheshmekaboodi, M., Georgiou, A., & Pirnejad, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01159-1 · Cited by 94 · see citing works
Sum, G., Nicholas, S. O., Nai, Z. L., Ding, Y. Y., & Tan, W. S. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03024-4 · Cited by 105 · see citing works
Astridge, B., Li, W. W., McDERMOTT, B. M., & Longhitano, C. (2023) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2023.106055 · Cited by 93 · see citing works
Arowosegbe, A., & Oyelade, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20021514 · Cited by 101 · see citing works
Hudon, C., Chouinard, M., Pluye, P., Sherif, R. E., Bush, P. L., Rihoux, B., Poitras, M.-È., Lambert, M., Zomahoun, H. T. V., & Légaré, F. (2019) · The Annals of Family Medicine · doi:10.1370/afm.2419 · Cited by 91 · see citing works
Self-citationGuemghar, I., Padilha, P. P. D. O., Abdel‐Baki, A., Jutras‐Aswad, D., Paquette, J., & Pomey, M. (2022) · JMIR Mental Health · doi:10.2196/36094 · Cited by 90 · see citing works
Afaya, A., Konlan, K. D., & Kim, H. O. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07187-5 · Cited by 89 · see citing works
“The third step detailed how the articles were assessed for rigor using the Mixed Method Appraisal Tool (MMAT) version 2018 [19].”
MethodMental health and psychological wellbeing in the early stages of doctoral study: a systematic review
Jackman, P. C., Jacobs, L., Hawkins, R. M., & Sisson, K. (2021) · European Journal of Higher Education · doi:10.1080/21568235.2021.1939752 · Cited by 88 · see citing works
“At least one quality concern was identified with more than half of studies in the review (see online Appendix 5 for additional information) based on the MMAT criteria (Hong et al. 2019).”
BackgroundMethodInfluentialCommunication Needs of Cancer Patients and/or Caregivers: A Critical Literature Review
Li, J., Luo, X., Cao, Q., Lin, Y., Xu, Y., & Li, Q. (2020) · Journal of Oncology · doi:10.1155/2020/7432849 · Cited by 87 · see citing works
“MMAT is a reliable and effective evaluation tool that can simultaneously evaluate qualitative, quantitative, and mixed methods studies [27, 28].”
Interventions to reduce low-value imaging – a systematic review of interventions and outcomes
Kjelle, E., Andersen, E. R., Soril, L. J. J., van Bodegom‐Vos, L., & Hofmann, B. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07004-z · Cited by 87 · see citing works
“The Mixed Methods Appraisal Tool (MMAT) was used to assess the methodical quality of all included studies as it is considered to be an appropriate tool for appraisal of interventional studies of different methodologies [34].”
MethodWestwell-Roper, C. Y., To, S., Andjelic, G., Lu, C., Lin, B., Soller, L., Chan, E. S., & Stewart, S. E. (2021) · Pediatric Allergy and Immunology · doi:10.1111/pai.13695 · Cited by 87 · see citing works
The Impact of AI Scribes on Streamlining Clinical Documentation: A Systematic Review
Sasseville, M., Yousefi, F., Ouellet, S., Naye, F., Stefan, T., Carnovale, V., Bergeron, F., Ling, L., Gheorghiu, B., Hagens, S., et al. (2025) · Healthcare · doi:10.3390/healthcare13121447 · Cited by 86 · see citing works
“The quality of each included review was independently assessed by pairs of reviewers using the Mixed Methods Appraisal Tool (MMAT) [28–30], with all scores verified by a senior reviewer.”
Establishing a clinical service to prevent psychosis: What, how and when? Systematic review
de Pablo, G. S., Estradé, A., Cutroni, M., Andlauer, O., & Fusar‐Poli, P. (2021) · Translational Psychiatry · doi:10.1038/s41398-020-01165-x · Cited by 83 · see citing works
Brassel, S., Power, E., Campbell, A., Brunner, M., & Togher, L. (2021) · Journal of Medical Internet Research · doi:10.2196/26344 · Cited by 84 · see citing works
Newington, L., Wells, M., Adonis, A., Bolton, L., Saghdaoui, L. B., Coffey, M., Crow, J., Costa, O. F. D., Hughes, C., Savage, M., et al. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06354-y · Cited by 82 · see citing works
“Outcomes of the MMAT assessment [20] are shown in Fig.”
MethodResultInfluentialHilton, N. Z., Addison, S., Ham, E., Rodrigues, N. C., & Seto, M. C. (2021) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12781 · Cited by 87 · see citing works
Knitter, B., & Zemp, M. (2020) · Digital Psychology · doi:10.24989/dp.v1i1.1809 · Cited by 81 · see citing works
Hughes, M. C., Liu, Y., & Baumbach, A. (2021) · Gerontology and Geriatric Medicine · doi:10.1177/23337214211020164 · Cited by 80 · see citing works
Moulaei, K., Sharifi, H. P., Bahaadinbeigy, K., & Dinari, F. (2024) · BMC Psychiatry · doi:10.1186/s12888-024-05563-z · Cited by 85 · see citing works
Baldassarre, A., Lulli, L. G., Cavallo, F., Fiorini, L., Mariniello, A., Mucci, N., & Arcangeli, G. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1039680 · Cited by 78 · see citing works
“The quality assessment was performed using the Mixed Methods Appraisal Tool (MMAT) (44, 45) which is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative, and mixed methods studies, thus, it was suitable for our review.”
MethodMörelius, E., Robinson, S., Arabiat, D. H., & Whitehead, L. (2021) · Journal of Medical Internet Research · doi:10.2196/31665 · Cited by 79 · see citing works
Moyle, W., Murfield, J., & Lion, K. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104513 · Cited by 75 · see citing works
Digital Storytelling for Health-Related Outcomes in Older Adults: Systematic Review
Stargatt, J., Bhar, S. S., Bhowmik, J., & Mahmud, A. A. (2021) · Journal of Medical Internet Research · doi:10.2196/28113 · Cited by 75 · see citing works
“Notably, the studies also did not conduct follow-up assessments to explore whether the effects were sustained for greater than a week [47] following digital storytelling.”
BackgroundWalker, R., Drakeley, S. C., Welch, R., Leahy, D., & Boyle, J. A. (2020) · Sex Education · doi:10.1080/14681811.2020.1843013 · Cited by 73 · see citing works
Therapeutic Use of Role-Playing Game (RPG) in Mental Health: A Scoping Review
Arenas, D. L., Viduani, A. C., & Araújo, R. B. (2022) · Simulation & Gaming · doi:10.1177/10468781211073720 · Cited by 73 · see citing works
“The quality of the studies was scored using a nominal scale and divided into: (1) qualitative studies; (2) randomized controlled trials (RCT); (3) non-randomized studies; (4) quantitative descriptive studies; and (5) mixedmethods studies (Hong et al., 2019).”
BackgroundMethodArtificial intelligence for diagnostics in radiology practice: a rapid systematic scoping review
Lawrence, R., Dodsworth, E., Massou, E., Sherlaw-Johnson, C., Ramsay, A. I. G., Walton, H., O'Regan, T., Gleeson, F., Crellin, N., Herbert, K., et al. (2025) · eClinicalMedicine · doi:10.1016/j.eclinm.2025.103228 · Cited by 69 · see citing works
McGuier, E. A., Kolko, D. J., Aarons, G. A., Schachter, A., Klem, M. L., Diabes, M. A., Weingart, L. R., Salas, E., & Wolk, C. B. (2024) · Implementation Science · doi:10.1186/s13012-024-01381-9 · Cited by 70 · see citing works
Hallett, N., Gayton, A., Dickenson, R., Franckel, M., & Dickens, G. L. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105845 · Cited by 73 · see citing works
Work environment of hospital nurses during the COVID‐19 pandemic in Brazil
dos Santos, J. L. G., Balsanelli, A. P., Freitas, E. D. O., Menegon, F. H. A., Carneiro, I. A., Lazzari, D. D., de Menezes, J. A. L., Erdmann, A. L., & Camponogara, S. (2021) · International Nursing Review · doi:10.1111/inr.12662 · Cited by 67 · see citing works
de Oliveira, J. L. C., de Magalhães, A. M. M., Matsuda, L. M., dos Santos, J. L. G., Souto, R. Q., Riboldi, C. D. O., & Ross, R. (2021) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2020-0603 · Cited by 66 · see citing works
Morisod, K., Malebranche, M. E., Marti, J., Spycher, J., Grazioli, V. S., & Bodenmann, P. (2022) · European Journal of Public Health · doi:10.1093/eurpub/ckac056 · Cited by 62 · see citing works
Leadership Development Strategies in Interprofessional Healthcare Collaboration: A Rapid Review
Bornman, J., & Louw, B. (2023) · Journal of Healthcare Leadership · doi:10.2147/jhl.s405983 · Cited by 71 · see citing works
Catsaros, S. M., & Wendland, J. (2020) · Midwifery · doi:10.1016/j.midw.2020.102666 · Cited by 61 · see citing works
“Quality assessment was perormed using the Mixed Methods Appraisal Tool (MMAT- Verion 18, Hong et al., 2019 ).”
MethodUse of Technology to Promote Health and Wellbeing of People Who Are Homeless: A Systematic Review
Heaslip, V., Richer, S., Simkhada, B., Doğan, H., & Green, S. M. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18136845 · Cited by 59 · see citing works
van der Velde, M., van der Leeden, M., Geleijn, E., Veenhof, C., & Valkenet, K. (2023) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-023-01474-6 · Cited by 60 · see citing works
The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Sports Medicine - Open · doi:10.1186/s40798-023-00655-8 · Cited by 59 · see citing works
Sleep disturbances in Lewy body dementia: A systematic review
Elder, G. J., Lázár, A. S., Alfonso‐Miller, P., & Taylor, J. (2022) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5814 · Cited by 56 · see citing works
Inocian, E. P., Hill, M. B., Felicilda‐Reynaldo, R. F. D., Kelly, S. H., Paragas, E. D., & Turk, M. T. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103391 · Cited by 56 · see citing works
Factors Impacting Retention of Aged Care Workers: A Systematic Review
Thwaites, C. K., McKercher, J. P., Fetherstonhaugh, D., Blackberry, I., Gilmartin‐Thomas, J., Taylor, N. F., Bourke, S. L., Fowler‐Davis, S., Hammond, S., & Morris, M. E. (2023) · Healthcare · doi:10.3390/healthcare11233008 · Cited by 56 · see citing works
Sams, L., Langdown, B. L., Simons, J., & Všetečková, J. (2023) · International Journal of Sports Physical Therapy · doi:10.26603/001c.73795 · Cited by 55 · see citing works
Alrawashdeh, G. S., Fyffe, S., Azevedo, R. F. L., & Castillo, N. M. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100587 · Cited by 53 · see citing works
Fisher, V., Li, W. W., & Malabu, U. H. (2023) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12441 · Cited by 53 · see citing works
Calo, M., Judd, B. K., & Peiris, C. L. (2024) · Medical Education · doi:10.1111/medu.15391 · Cited by 52 · see citing works
“39”
Overview of the role of robots in upper limb disabilities rehabilitation: a scoping review
Moulaei, K., Bahaadinbeigy, K., Haghdoostd, A., Nezhad, M. S., & Sheikhtaheri, A. (2023) · Archives of Public Health · doi:10.1186/s13690-023-01100-8 · Cited by 51 · see citing works
Collins, J., & Murphy, G. H. (2021) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12954 · Cited by 50 · see citing works
Hasnan, S., Aggarwal, S., Mohammadi, L., & Koczwara, B. (2022) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2022.06.004 · Cited by 50 · see citing works
Nurses knowledge and attitude regarding pain: A systematic review and meta-analysis
Ortiz, M. I., Cuevas‐Suárez, C. E., Cariño‐Cortés, R., Navarrete-Hernández, J. D. J., & González-Montiel, C. A. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103390 · Cited by 52 · see citing works
Rodrigues, D. A., Fiol-deRoque, M. A., Mateos-Campos, R., Figueiras, A., Herdeiro, M. T., & Roque, F. (2024) · Digital Health · doi:10.1177/20552076231225133 · Cited by 59 · see citing works
“34,35 All discrepancies were resolved through discussion with a third researcher (MTH, AF or FR).”
BackgroundSmart villages concept — A bibliometric analysis and state-of-the-art literature review
Bokun, K., & Nazarko, J. (2023) · Progress in Planning · doi:10.1016/j.progress.2023.100765 · Cited by 49 · see citing works
Parental attitudes towards mandatory vaccination; a systematic review
Smith, L., Hodson, A., & Rubin, G. J. (2021) · Vaccine · doi:10.1016/j.vaccine.2021.06.018 · Cited by 49 · see citing works
Graduates’ employability skills in East Africa
Ortiz, L. G., Mancini, F., Jacobetty, P., & Maina, M. F. (2021) · Journal of Teaching and Learning for Graduate Employability · doi:10.21153/jtlge2021vol12no2art988 · Cited by 50 · see citing works
A Comprehensive Analysis of Factors Associated with Intimate Partner Femicide: A Systematic Review
Garcia-Vergara, E., Almeda, N., Ríos, B. M., Becerra‐Alonso, D., & Fernández‐Navarro, F. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19127336 · Cited by 53 · see citing works
Li, W. W., Heward, C., Merrick, A., Astridge, B., & Leow, T. Y. Q. (2024) · Journal of Pacific Rim Psychology · doi:10.1177/18344909241252325 · Cited by 48 · see citing works
“The full-text assessment using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2019) was the second step for study selection and was undertaken by three authors (WL, BA, and AM).”
The Effectiveness of e-Health Solutions for Aging With Cognitive Impairment: A Systematic Review
Dequanter, S., Gagnon, M., Ndiaye, M., Gorus, E., Fobelets, M., Giguere, A. M. C., Bourbonnais, A., & Buyl, R. (2020) · The Gerontologist · doi:10.1093/geront/gnaa065 · Cited by 48 · see citing works
Self-citationHomer, C. S., Turkmani, S., Wilson, A. N., Vogel, J. P., Shah, M. G., Fogstad, H., & Langlois, E. V. (2022) · BMJ Global Health · doi:10.1136/bmjgh-2021-006872 · Cited by 48 · see citing works
“30 Patient and public involvement Patients and the public were not directly involved in this review.”
BackgroundWolff, A. C., Dresselhuis, A., Hejazi, S., Dixon, D., Gibson, D., Howard, A. F., Liva, S., Astle, B., Kirkham, S. R., Noonan, V. K., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01725-2 · Cited by 47 · see citing works
Digital Technologies to Prevent Social Isolation and Loneliness in Dementia: A Systematic Review
Kaur, H., Kernaghan, D., Schoonmade, L. J., Egan, K. J., & Pot, A. M. (2022) · Journal of Alzheimer s Disease · doi:10.3233/jad-220438 · Cited by 46 · see citing works
Gunn, V., Kreshpaj, B., Santander, N. M., Vignola, E. F., Wegman, D. H., Hogstedt, C., Ahonen, E. Q., Bodin, T., Orellana, C., Baron, S., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042232 · Cited by 49 · see citing works
“The MMAT tool is not as detailed and extensive as other tools designed for qualitative and quantitative studies, such as the Critical Appraisal Skills Programme (CASP) Checklists [64], but its combination of tested usefulness, reliability, and ease of application with many heterogenous studies made it a robust choice…”
MethodFàbregues, S., Sáinz, M., Romano, M. J., Escalante‐Barrios, E. L., Younas, A., & Pérez, B. S. L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.956300 · Cited by 45 · see citing works
Self-citationNursing pre-licensure and graduate education for LGBTQ health: A systematic review
Sherman, A. D. F., Smith, S. K., Moore, S. E., Coleman, C. L., Hughes, T. L., Dorsen, C. G., Balthazar, M. S., Klepper, M., Mukerjee, R., & Bower, K. M. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2022.12.003 · Cited by 45 · see citing works
Volf, K., Kelly, L. P., Bengoechea, E. G., Casey, B., Gelius, P. C., Messing, S., Forberger, S., Lakerveld, J., den Braver, N. R., Żukowska, J., et al. (2022) · International Journal of Sport Policy and Politics · doi:10.1080/19406940.2022.2127835 · Cited by 46 · see citing works
Zhou, F., Zhang, H., Wang, H. Y., Liu, L. F., & Zhang, X. (2024) · Aging Clinical and Experimental Research · doi:10.1007/s40520-023-02652-z · Cited by 44 · see citing works
“The Mixed Methods Appraisal Tool (MMAT) is used by two independent assessors to rigorously evaluate the quality of included studies [31].”
MethodClozapine prescribing barriers in the management of treatment-resistant schizophrenia
Baig, A. I., Bazargan‐Hejazi, S., Ebrahim, G., & Rodriguez-Lara, J. (2021) · Medicine · doi:10.1097/md.0000000000027694 · Cited by 44 · see citing works
“Data quality and risk of bias in individual studies The first 2 authors critically assessed and graded the studies for risk of bias using the Mixed Methods Appraisal Tool (MMAT),([13]) and the Measurement Tool to Assess Systematic Reviews 2.”
MethodBreast cancer screening in sub-Saharan Africa: a systematic review and ethical appraisal
Martei, Y. M., Dauda, B., & Vanderpuye, V. D. N. K. (2022) · BMC Cancer · doi:10.1186/s12885-022-09299-5 · Cited by 43 · see citing works
“The mixed methods appraisal tool (MMAT) [17] was used to assessed the quality of the studies included in the analysis.”
Method“Wire Syndrome” Following Bonded Orthodontic Retainers: A Systematic Review of the Literature
Charavet, C., Vives, F., Aroca, S., & Dridi, S. (2022) · Healthcare · doi:10.3390/healthcare10020379 · Cited by 43 · see citing works
“Risk of Bias Analysis Given the heterogeneity of the studies selected for this SRL, a tool that can evaluate the methodological quality of different types of studies within the same SRL was employed: The “Mixed Methods Appraisal Tool” (MMAT) [9] (http://mixedmethodsappraisaltoolpublic. pbworks.com accessed on 1 Septem…”
MethodGaupholm, J., Papadopoulos, A., Asif, A., Dodd, W., & Little, M. (2022) · Nutrition and Health · doi:10.1177/02601060221112810 · Cited by 43 · see citing works
“The latest version of the Mixed Methods Appraisal Tool (MMAT) was used for quality assessment (Hong et al., 2019).”
MethodToward a Framework for Appraising the Quality of Integration in Mixed Methods Research
Fàbregues, S., Younas, A., Escalante‐Barrios, E. L., Molina-Azorín, J. F., & Vázquez-Miraz, P. (2024) · Journal of Mixed Methods Research · doi:10.1177/15586898241257555 · Cited by 43 · see citing works
“Second, the MMIQF does not limit the appraisal to a single type of quality as it combines methodological and reporting quality criteria (Hong & Pluye, 2019).”
Self-citationZamir, A., Tickle, A., & Sabin‐Farrell, R. (2022) · Journal of Clinical Psychology · doi:10.1002/jclp.23339 · Cited by 42 · see citing works
Murtagh, E., Calderón, A., Scanlon, D., & MacPhail, A. (2023) · European Physical Education Review · doi:10.1177/1356336x231155793 · Cited by 34 · see citing works
“The MMAT criteria focus on the core criteria that may hinder the validity of the findings of a study, for example, data collection methods, outcome data, and interpretation of results (Hong et al., 2019).”
BackgroundMotyer, G., Dooley, B. A., Kiely, P., & Fitzgerald, A. (2020) · Patient Education and Counseling · doi:10.1016/j.pec.2020.11.023 · Cited by 43 · see citing works
Collins, J., Horton, K., Ives, E. G.-S., Murphy, G. H., & Barnoux, M. F. L. (2022) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-022-05590-3 · Cited by 45 · see citing works
Khosravipour, M., & Khosravipour, H. (2020) · Chemosphere · doi:10.1016/j.chemosphere.2019.125680 · Cited by 41 · see citing works
Leow, T. Y. Q., Li, W. W., Miller, D. J., & McDERMOTT, B. M. (2024) · Journal of Mental Health · doi:10.1080/09638237.2024.2332812 · Cited by 40 · see citing works
“Articles that were disputed were discussed to achieve consensus about inclusion in the second step: the methodological appraisal of the full-text articles employing the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2019).”
Protocol for a systematic review of living labs in healthcare
Archibald, M. M., Wiebe, S., Rieger, K. L., Linton, J., & Woodgate, R. L. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-039246 · Cited by 39 · see citing works
Davies, J. F., Ikin, B., Francis, J., & McGain, F. (2023) · British Journal of Anaesthesia · doi:10.1016/j.bja.2023.05.017 · Cited by 39 · see citing works
Alipour, J., & Haghighi, M. H. H. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735181 · Cited by 40 · see citing works
Effects of Air Pollution on the Health of Older Adults during Physical Activities: Mapping Review
Andrade, A., D’Oliveira, A., de Souza, L. C., Bastos, A. C. R. D. F., Dominski, F. H., Stabile, L., & Buonanno, G. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043506 · Cited by 38 · see citing works
“The reliability and validity of the new version of the MMAT have been confirmed [30,31].”
Güler, A., Lee, R. C., Rojas‐Guyler, L., Lambert, J., & Smith, C. R. (2023) · Nursing Inquiry · doi:10.1111/nin.12589 · Cited by 38 · see citing works
- Authors
- Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., & Vedel, I.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 111
- Pages
- 49-59.e1
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2924322406
- Cited by
- 1,082
- References
- 50
- Citations by year
- 2026: 205 · 2025: 178 · 2024: 190 · 2023: 174 · 2022: 163 · 2021: 110 · 2020: 57 · 2019: 5
- FWCI
- 168.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- The Oberta in Open Access (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Health Education and Validation
- Keywords
- Mixed Methods Appraisal Tool, content validity, critical appraisal, methodological quality criteria, qualitative research, survey research, mixed methods, e-Delphi, expert consensus
- MeSH terms
- Systematic Reviews as Topic, Delphi Technique, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
