The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,701–1,800 of 4,578
Systematic review of factors influencing decisions to limit treatment in the emergency department
Walzl, N., Sammy, I. A., Taylor, P., Smith, J. E., & Lowe, D. J. (2021) · Emergency Medicine Journal · doi:10.1136/emermed-2019-209398 · Cited by 11
The Acceptance of Human Papillomavirus Self-Sampling Test among Muslim Women:A Systematic Review
Amir, S. M., Idris, I., & Yusoff, H. M. (2022) · Asian Pacific Journal of Cancer Prevention · doi:10.31557/apjcp.2022.23.3.767 · Cited by 9
“The Mixed Methods Appraisal Tool (MMAT) was used next, to appropriately evaluate the quality of the selected articles for reviewing (Hong et al., 2018).”
MethodNursing students’ experiences of seeking mental health support: a literature review
Morton, C. (2022) · Mental Health Practice · doi:10.7748/mhp.2022.e1604 · Cited by 11
Impact of nurse-surgeons on patient-centred outcomes: A systematic review
Grota, T., Betihavas, V., Burston, A., & Jacob, E. R. (2022) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2022.100086 · Cited by 12
“The Mixed Methods Appraisal Tool (Hong et al., 2018) was used to assess the quality of each study.”
BackgroundMethodInfluentialTunnard, I., Gillam, J., Harvey, C., Davies, N., Vickerstaff, V., Ellis-Smith, C. I., & Evans, C. (2022) · Frontiers in Dementia · doi:10.3389/frdem.2022.977561 · Cited by 6
“ACP, Advanced Care Planning; BPSD, Behavioral and Psychological Symptoms of Dementia; CASP, Critical Appraisal Skills Programme; CDR, Clinical Dementia Rating; CVA, Cerebrovascular accident; EPR, Electronic Patient Record; GP, General Practitioner; interRAI-LTCF, International Resident Assessment Instrument for Long-T…”
MethodInfluentialHarmanci, D., Edelman, N. L., Richardson, D., Lunt, A., & Llewellyn, C. D. (2023) · International Journal of Adolescent Medicine and Health · doi:10.1515/ijamh-2022-0090 · Cited by 8
Hu, X., Ren, J., Wang, P., Chen, W., Shen, W., Li, Y., Xiao, T., Dai, Y., & Sang, Y. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2200261 · Cited by 10
“The Mixed Methods Appraisal Tool (MMAT) 2018 version was used by two independent raters (YS and YD) to assess the methodological quality of included studies [45].”
BackgroundMethodInfluentialExperiences with Cling Film and Dental Dam Use in Oral Sex: A Mixed-Methods Systematic Review
Kanmodi, K. K., Egbedina, E. A., Nkhata, M. J., & Nnyanzi, L. A. (2023) · Oral · doi:10.3390/oral3020019 · Cited by 9
“, 2018) [57] Total Score (Over 7) Grading Status Screening Questions Questions Specific to Study Design”
BackgroundSahoo, K. C., Negi, S., Mahapatra, P., Samantaray, K., Dash, G. C., Dubey, S., Sahay, M. R., Sahoo, R. K., Bhattacharya, D., Sahoo, B., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1170386 · Cited by 13
“We employed the mixed-method-appraisal tool (MMAT) to assess their quality (22) (Appendix II).”
MethodPsychological consequences of female genital mutilation: A mixed-method systematic review
Reman, T., Balligand, V., Schoefs, B., Feipel, V., & Bertuit, J. (2023) · South African Journal of Physiotherapy · doi:10.4102/sajp.v79i1.1877 · Cited by 8
“It is a validated and reliable tool that offers criteria specific to observational and interventional study design, including quantitative and qualitative designs (Hong et al. 2018; Pace et al. 2012).”
MethodAutistic People and Telehealth Practice During the COVID-19 Pandemic: a Scoping Review
Bundy, R., Mandy, W., Kenny, L., & Ali, D. (2023) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-023-00387-1 · Cited by 10
“We used the mixed methods appraisal tool (MMAT) (Table 3) (Hong et al., 2018) to evaluate methodological rigour.”
Self-management Behaviors and Associated Factors in Adult Cancer Survivors: An Integrative Review
Aung, M. L., & Cheng, H. (2023) · Cancer Nursing · doi:10.1097/ncc.0000000000001289 · Cited by 10
Intragroup conflict in sports: a scoping review
Lüdemann, J., & Kleinert, J. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2023.2278135 · Cited by 14
“According to the inclusion criteria, the MMAT sets of criteria for quantitative studies (randomized controlled trials, non-randomized studies, and quantitative descriptive studies) were considered for this review (Hong et al., 2018).”
Bartrim, K., Waite, C., Williams, L. T., & O'Shea, M. (2023) · Nutrition & Dietetics · doi:10.1111/1747-0080.12856 · Cited by 11
Exercise Motivation among Fitness Center Members: A Combined Qualitative and Q-Sorting Approach
Nam, T. T., Suwanbamrung, C., Dadras, O., Hou, X., Phu, D. H., Tin, P. T., Quang, H. K., Thuy, T. T. N., Ageru, T. A., Aekwarankoon, S., et al. (2023) · Journal of Human Earth and Future · doi:10.28991/hef-2023-04-03-07 · Cited by 9
“Specifically, the Critical Appraisal Skills Programme (CASP) checklist for appraisal of qualitative research (version 2018) was utilized to guide the evaluation process, serving the needs of information professionals and researchers [30, 31].”
MethodSubran, S., & Mahmud, S. N. D. (2024) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v13-i1/20455 · Cited by 20
“Source: (Hong et al., 2018) From the 29 selected articles, a diverse array of research designs was identified.”
BackgroundNtho, T. A., Themane, M. J., Sepadi, M. D., Phochana, T. S., Sodi, T., & Quarshie, E. N. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-080675 · Cited by 13
The mental health and well-being implications of a mesothelioma diagnosis: A mixed methods study
Sherborne, V., Wood, E., Mayland, C. R., Gardiner, C., Lusted, C., Bibby, A. C., Tod, A., Taylor, B. H., & Ejegi‐Memeh, S. (2024) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2024.102545 · Cited by 8
Schüttengruber, G., Olsson, M. M., Holmberg, C., Großschädl, F., Hessman, E., Sjöblom, H., & Heckemann, B. (2024) · Geriatric Nursing · doi:10.1016/j.gerinurse.2024.05.017 · Cited by 12
Comber, E. R., Keogh, S. J., Nguyen, L. N., Byrnes, J., & Ullman, A. J. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16268 · Cited by 11
“Empirical studies were assessed for risk of bias using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018 ).”
Resilience in adolescents with type 1 diabetes: An integrative review
Survonen, A., Suhonen, R., & Joronen, K. (2024) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2024.06.007 · Cited by 8
Building a learning health care community in rural and remote areas: a systematic review
Asghari, S., Bent, J. E., Modir, A., MacDonald, A., Farrell, A., Bethune, C., & Graham, W. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11194-7 · Cited by 14
“The quality of each study was assessed independently by two authors using the Mixed Methods Appraisal Tool (MMAT) scoring system [15].”
MethodNathavitharana, R. R., Pearl, A., Biewer, A., Young, L., Mukasa, L., Delrooz, N., Subramanian, A., Miller, S., Mase, S., Munsiff, S. S., et al. (2024) · Clinical Infectious Diseases · doi:10.1093/cid/ciae496 · Cited by 8
Gibbs, V., Cai, R. Y., Love, A. M. A., Edwards, C., & Ansari, M. A. (2024) · Autism in Adulthood · doi:10.1089/aut.2024.0272 · Cited by 13
What patients want from access to UK general practice: systematic review
Atherton, H., Leach, H., Mortell, R., & Parsons, J. (2025) · British Journal of General Practice · doi:10.3399/bjgp.2024.0582 · Cited by 11
“It provides a thorough quality assessment by assessing each study according to the study design and provides a validated way to compare different study designs in relation to quality, 20 without biasing the process for or against any given study design.”
Kalideen, L., Govender, P., Wyk, J. V., & Kuupiel, D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01709-2 · Cited by 8
“To facilitate this, this study will use the mixed method quality assessment tool [30].”
MethodGasteiger, N., van der Veer, S. N., Wilson, P. M., & Dowding, D. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw55335.2022.00218 · Cited by 11
Effekte von Maßnahmen des Spielerschutzes beim Online-Glücksspiel
Hayer, T., & Kalke, J. (2022) · SUCHT - Zeitschrift für Wissenschaft und Praxis / Journal of Addiction Research and Practice · doi:10.1024/0939-5911/a000769 · Cited by 9
Patient and carer access to medicines at end of life: the ActMed mixed-methods study
Latter, S. M., Campling, N. C., Birtwistle, J. C., Richardson, A. M., Bennett, M., Meads, D. M., Blenkinsopp, A., Breen, L., Edwards, Z., Sloan, C., et al. (2022) · Health and Social Care Delivery Research · doi:10.3310/fiqe5189 · Cited by 13
Jafree, S. R., Mahmood, Q. K., Sohail, M. M., Asim, M., & Barlow, J. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061644 · Cited by 8
Smith, D. J., Mizelle, E., Leslie, S., Li, G. X., Stone, S., Stauffer, P., Smith, A., Lewis, G., Rodden, E. L., McDermott‐Levy, R., et al. (2023) · Environmental Research Health · doi:10.1088/2752-5309/acbbe6 · Cited by 9
“The tool can be used to appraise five different study designs (qualitative, randomized controlled, nonrandomized, quantitative descriptive, and mixed methods) and includes 25 criteria and two screening questions [20].”
MethodInfluentialFakolade, A. O., Akbar, N., Mehelay, S., Phadke, S., Tang, M., Alqahtani, A., Pullattayil, A. K., & Busse, M. E. (2023) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2023.104606 · Cited by 10
Specialist dementia nursing models and impacts: a systematic review
Brown, P., Burton, A., Ayden, J., Dening, K. H., Hoe, J., & Cooper, C. (2023) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000874 · Cited by 9
“We sought to prioritise evidence from higher quality papers, and used the Mixed Methods Appraisal Tool, (MMAT) to assess study quality [6], Findings were then discussed and shared to check for consistency.”
MethodFactors associated with occupational asthma among food industry workers: A systematic review
Zamri, A. S. S. M., Saruddin, M. Z., Harun, A., Aziz, S. F. A., Za’bah, A. K. A., Dapari, R., Hassan, M. R., Dom, N. C., & Rahim, S. S. S. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287040 · Cited by 11
Yadav, U. N., Smith, M. L., Agostino, J., Sinka, V., Williamson, L. M., Wyber, R., Butler, D., Belfrage, M. K., Freeman, K., Passey, M. E., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-071234 · Cited by 10
Pitt, E., Bradford, N., Robertson, E. G., Sansom‐Daly, U. M., & Alexander, K. E. (2023) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2023.102398 · Cited by 12
Li, W., Su, M., Zhang, W., Fan, X., Li, R., Gao, Y., & Wei, X. (2023) · Health Research Policy and Systems · doi:10.1186/s12961-023-01054-x · Cited by 10
“Moreover, they will rank the quality of the selected articles as low, moderate or high on the basis of the MMAT criteria [51].”
BackgroundJeyasingh, B. B. F. (2023) · Digital Economy and Sustainable Development · doi:10.1007/s44265-023-00024-z · Cited by 19
“If the screening questions in the MMAT appraisal tool had a response of “No” or “Can’t tell,” it was deemed that further appraisal was not appropriate (Hong et al. 2018, p. 5).”
MethodDetermining the period of communicability of SARS-CoV-2: A rapid review of the literaturePreprint
Park, M., Pawliuk, C., Nguyen, T., Griffitt, A., Dix‐Cooper, L., Fourik, N., & Dawes, M. G. (2020) · medRxiv · doi:10.1101/2020.07.28.20163873 · Cited by 8
“Assessment of study quality: An adaptation of the Mixed Methods Appraisal Tool was used to assess study quality [10].”
MethodLi, W., Su, M., Li, Z., & Fan, X. (2025) · Health Research Policy and Systems · doi:10.1186/s12961-024-01277-6 · Cited by 8
“W.L. and M.S. will independently use the MMAT to evaluate the methodological quality of each category of the included studies, providing responses of “Yes,” “No” or “Can’t tell” [51].”
InfluentialNedergaard, H. K., Duch, P., Wikkelso, A. J., Abildgren, M. L., Lund-Olesen, J., Wildgaard, K., & Ekelund, K. (2026) · International Journal of Obstetric Anesthesia · doi:10.1016/j.ijoa.2026.104882 · Cited by 8
Social Media Use Among Parents and Caregivers of Children With Rare Genetic Diseases: Scoping Review
Davis, C., Bogaert, L., Powell, J., & Low, K. (2025) · Journal of Medical Internet Research · doi:10.2196/77087 · Cited by 8
Gebran, E., Antoniou, M., Samtani, S., Brookman, R., & Siette, J. (2025) · Dementia · doi:10.1177/14713012251412051 · Cited by 7
Ergen Sporcularda Spora Bağlılık İle İlişkili Bireysel ve Sosyal Faktörler: Sistematik Derleme
MERDAN, H. E., & Çağlar, E. (2022) · Spor Bilimleri Dergisi · doi:10.17644/sbd.1008784 · Cited by 15
Meadows, T., & Davey, G. (2022) · Leprosy Review · doi:10.47276/lr.93.4.311 · Cited by 7
Clibbens, N., Booth, A., Sharda, L., Baker, J. A., Thompson, J., Ashman, M., Berzins, K. M., Weich, S., & Kendal, S. (2023) · International Journal of Mental Health Nursing · doi:10.1111/inm.13204 · Cited by 7
Brunt, L. J., Rivalland, A., Panek‐Hudson, Y., & Krishnasamy, M. (2023) · Cancer Nursing · doi:10.1097/ncc.0000000000001292 · Cited by 6
Gulyani, P., Rawat, P., Elmi, Y., Gupta, S., & Wan, C. S. (2024) · Diabetes & Metabolic Syndrome Clinical Research & Reviews · doi:10.1016/j.dsx.2024.102944 · Cited by 11
“…Methods Appraisal Tool (MMAT) designed to appraise the methodological quality of five types of study designs (which were qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies and mixed-methods studies) in a mixed-methods systematic review [24].”
Informal coercion in inpatient mental healthcare: a scoping review protocol
Billé, V., Lessard‐Deschênes, C., Sergerie‐Richard, S., Désilets, M., Tansey, J., & Goulet, M. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-079549 · Cited by 10
“…a critical appraisal and evaluation of the quality of the literature will be applied using the following tools: the Mixed Methods Appraisal Tool, 28 the checklist for systematic reviews and research synthesis 29 and the Authority, Accuracy, Coverage, Objec-tivity, Date, Significance checklist…”
MethodRodríguez-Martín, S., Greig, Y. M., Shaw, E., McKellar, L. V., & Kuipers, Y. (2024) · Nurse Education in Practice · doi:10.1016/j.nepr.2024.103995 · Cited by 11
Bassey, A. E., & Zaka, R. (2024) · Public Health Challenges · doi:10.1002/puh2.181 · Cited by 14
“Hence, to critically appraise these studies, two separate standardized tools were employed: the Critical Appraisal Skills Program (CASP) qualitative checklist and the Mixed Methods Appraisal Tool (MMAT) Version 2018 [14], as CASP tools do not include a mixed-methods checklist [15].”
Park, S. Y., Kim, Y., Katapodi, M. C., Kim, Y.-J., Chae, H., Choi, Y., Ryu, K. H., Lee, E., Kong, S., & Jung, S. (2024) · Cancers · doi:10.3390/cancers16111963 · Cited by 7
A Scoping Review of Trauma, Mental Health and First Responders in Australia
Jones, R., Jackson, D. W., Ranse, J., Arena, A. F. A., Clegg, L., Sutton, C., Gayed, A., Rice, K., & Usher, K. J. (2024) · International Journal of Mental Health Nursing · doi:10.1111/inm.13397 · Cited by 10
Ahmed, H., Bendall, C., Anwar, F., Al‐Janabi, M., & Wood, L. (2024) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.3032 · Cited by 7
“The quality of the studies ranged from moderate to high according to MMAT criteria (Hong et al. 2018).”
InfluentialThobane, K. F., Mulaudzi, F. M., Lavhelani, N. R., & Gundo, R. (2024) · International Journal of Africa Nursing Sciences · doi:10.1016/j.ijans.2024.100761 · Cited by 10
Crying wolf, alarm safety and management in paediatrics: A scoping review
Cole, R., Roderick, G., Cheema, O., Cunninghame, J., & Ullman, A. J. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16398 · Cited by 7
“…pro - cess of study quality critique due to the exploratory and descriptive nature of scoping reviews (Peters et al., 2020 ) the mixed methods appraisal tool (MMAT) was utilized to determine the quality of the eligible studies to increase the robustness of this scoping review (Hong et al., 2018 ).”
The factors associated with paediatric medical post-traumatic stress: A systematic review
Marcev, I., Lannon-Boran, C., Hyland, P., & Power, J. M. (2024) · Journal of Health Psychology · doi:10.1177/13591053241272214 · Cited by 9
“Of Interventions) was used for meta-analyses (Sterne et al., 2016) and the MMAT (Mixed Methods Appraisal Tool; Hong et al., 2018) was used on the quantitative and qualitative studies included in the review.”
Zhang, X., & Wu, Y. (2024) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108455 · Cited by 10
Cultural adaptations of the WHO iSupport for dementia: A scoping review
Sani, T. P., Cheung, G., Peri, K., Yates, S., Kerse, N. M., Whaanga, H., & Cullum, S. J. (2024) · Dementia · doi:10.1177/14713012241283860 · Cited by 12
“We performed a quality assessment of the studies ’ methodologies with the Mixed Method Appraisal Tool (MMAT version 2018), which was developed to appraise studies included in systematic mixed studies reviews (Hong et al., 2018).”
Greening the workforce: a systematic literature review of determinants in green HRM
Bimo, I. D., & Sulistyaningsih, E. (2024) · Cogent Business & Management · doi:10.1080/23311975.2024.2429793 · Cited by 30
“The research utilized the Mixed-Methods Appraisal Tool (MMAT) developed by Hong et al. (2018) to ensure rigorous quality control and a robust review process.”
The mental health outcomes of driving cessation for older people: A systematic review
Liddle, J., Scott, T., Hoyle, M. F., Salmon, A. J., Gustafsson, L. A., & Pachana, N. A. (2024) · International Psychogeriatrics · doi:10.1016/j.inpsyc.2024.100020 · Cited by 12
Bolarinwa, O. A., Odimegwu, C. O., Ajayi, K. V., Oni, T. O., Sah, R. K., & Akinyemi, A. I. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-12028-2 · Cited by 9
“…assessment of 30 selected eligible articles was independently assessed by two authors (OAB & KA) using the mixed Methods Appraisal Tool (MMAT) [43] to ensure that all included studies were of all high-quality and where there is a discrepancy in the quality of the study to be included or…”
Kohl, G., Ulate, M. M., Bhatt, J., Lynch, J. K., Scior, K., & Charlesworth, G. M. (2024) · The Gerontologist · doi:10.1093/geront/gnae186 · Cited by 7
Pedersen, M. L., Bricca, A., Baker, J. A., Schjerning, O., Munk‐Olsen, T., & Gildberg, F. A. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2024-301399 · Cited by 7
Development of mobile educational applications designed for nurses: A narrative review
Armour, T., Coffey, E. K., Manias, E., Redley, B., & Nicholson, P. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106576 · Cited by 13
Carer Involvement in Rehabilitation for People Living With Dementia: A Systematic Review
Lee, C., Jeon, Y., & Watson, K. M. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.16815 · Cited by 9
“…independently assessed the risk of bias and the quality of included articles based on the Cochrane risk- of-bias Tool version 1 (RoB) (Higgins and Green 2011) and Mixed Methods Appraisal Tool (MMAT) (Hong et al. 2018), and consulted with the remaining authors (Y-HJ and KW) for any disagreements.”
InfluentialScripps, E., Sutherland, D., Langdon, P. E., Hastings, R. P., & Gray, K. M. (2025) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.70004 · Cited by 14
“The Mixed Methods Appraisal Tool (MMAT, Hong et al., 2018) was used to assess methodological quality by providing questions for all study designs and generating a total MMAT score (Table 9).”
Graham, M., Hodgson, P., Fleming, L., Innerd, A. L., Clibbens, N., Hope, W., Aston, L., & Glascott, M. (2025) · International Journal of Mental Health Nursing · doi:10.1111/inm.70017 · Cited by 10
“The Mixed Method Appraisal Tool (MMAT) (Hong et al. 2018) was used to appraise and extract data on the quality of each included study.”
Beyond BRCA: A scoping review of person-centred care for women diagnosed with a BRCA gene mutation
Witte, L. E., van Diepen, C., & Cramm, J. M. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-094416 · Cited by 10
Analysis for health system resilience against the economic crisis: a best-fit framework synthesis
Foroughi, Z., Ebrahimi, P., Yazdani, S., Aryankhesal, A., Heydari, M., & Maleki, M. (2025) · Health Research Policy and Systems · doi:10.1186/s12961-025-01285-0 · Cited by 15
“Studies were appraised and scored on a five-point scale (0, 25, 50, 75 and 100) by two authors (ZF, MM) [61, 62].”
Psychosocial Problems in People Living with Thalassemia: A Systematic Review
Wangi, K. Y. W., Shaleha, R. R. A., Wijaya, E., & Birriel, B. (2025) · SAGE Open Nursing · doi:10.1177/23779608251323811 · Cited by 15
“The criteria rated “ no ” is equal to “ cannot tell ” (Hong et al., 2018), and these scored as 0.”
InfluentialAdverse childhood experiences and interpersonal functioning in adulthood: A systemic review
Maloney, E., Dowling, C., & O’Reilly, G. (2025) · European Journal of Trauma & Dissociation · doi:10.1016/j.ejtd.2025.100534 · Cited by 13
A Systematic Review of the Methods Used in Rapid Approaches to Research and Evaluation
Clark, S. E., Juan, N. V. S., Moniz, T., Appleton, R., Barnett, P., & Vindrola‐Padros, C. (2025) · Evaluation Review · doi:10.1177/0193841x251338536 · Cited by 7
“The MMAT allows the quality of heterogeneous study designs to be assessed based on the methodology used.”
InfluentialOzucelik, Y., Collins, J. C., & Pace, J. (2025) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2025.102418 · Cited by 7
Noselli, M., van der Vegt, A. H., Cordeil, M., Campbell, V. K., Wang, A. P., Shetty, A., & Scott, I. A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3572494 · Cited by 6
“Given the wide range of study designs, the Mixed Method Appraisal Tool (MMAT) was employed for study quality assessments [55].”
Giannakopoulos, G., & Prassou, A. (2025) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe15060098 · Cited by 17
The psychological impact of spending a prolonged time awaiting asylum
Shahzad, A., Katona, C., & Glover, N. (2025) · European Journal of Psychotraumatology · doi:10.1080/20008066.2025.2506189 · Cited by 8
“Another significant limitation is that many studies scored poorly against the MMAT, highlighting short-comings in the evidence base which requires thorough investigation.”
InfluentialChen, L. Y., Yang, L.-C., Lin, F.-Y., & Lin, Y.-H. P. (2025) · Archives of Psychiatric Nursing · doi:10.1016/j.apnu.2025.151916 · Cited by 10
Fisher, E., Chick, I., Fossey, J., & Spector, A. E. (2025) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.70124 · Cited by 7
“We used the Mixed Methods Appraisal Tool (MMAT) for mixed methods studies [47].”
Witziers, M., Looijmans, A., Jia, N., & Hagedoorn, M. (2025) · Journal of cancer survivorship · doi:10.1007/s11764-025-01866-6 · Cited by 8
Exit strategies for health interventions in low- and middle-income countries: a systematic review
Milenova, M., Thusini, S., Gronholm, P. C., & Salisbury, T. T. (2025) · BMC Global and Public Health · doi:10.1186/s44263-025-00182-6 · Cited by 11
“Data extraction and synthesis were carried out using Microsoft Word and Microsoft Excel following established procedures [29].”
Scoping Review: The Effectiveness of Interprofessional Virtual Reality Simulation
Alghanaim, N., Hart, J., & Finn, G. M. (2025) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205251346327 · Cited by 11
“60 MMAT was the only appraisal tool employed for mixed-method studies.”
Safarifard, R., Kiernan, G., Corcoran, Y., Courtney, E., Mitchell, J., Akard, T. F., & Lambert, V. (2025) · Palliative Medicine · doi:10.1177/02692163251353006 · Cited by 9
“36 Studies were assessed using these criteria, ensuring a rigorous evaluation of their methodological quality.”
Perera, A., Samtani, S., Karamacoska, D., & Siette, J. (2025) · Australian Journal on Ageing · doi:10.1111/ajag.70071 · Cited by 7
“27 The MMAT provides a comprehensive assessment of qualitative, quantitative and mixed-methods studies by employing specific screening criteria tailored to each study type.”
Sorrentino, M., Mercogliano, M., Fiorilla, C., Stilo, I., Esposito, F., Moccia, M., Lavorgna, L., Affinito, G., Salvatore, E., Sormani, M. P., et al. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1605993 · Cited by 12
Effects of tourism on well-being from the perspective of key actors: a scoping review
Vásquez, D. E. G., Rincón, A. G., & Vitery, F. C. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05792-x · Cited by 19
“Regarding the quality assessment performed, 72% of the articles meet the NMAT (Hong et al. 2018) and GRADE-CERQual (Lewin et al. 2018) criteria, while 28% present some quali fi cation in at least 1 of the 5 aspects of each type of design evaluated.”
Huang, M., Cheong, N., Zhang, Z., & Liu, J. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3610636 · Cited by 8
“2 (cid:2) Mixed Methods: Mixed Methods Appraisal Tool (MMAT) v.2018 [12]; (cid:2) Case Studies: Stake’s quality criteria for case study research [13].”
Perkins, A., Michalek, J. E., Dikomitis, L., Shergill, S. S., & Mareschal, I. (2025) · The British Journal of Psychiatry · doi:10.1192/bjp.2025.10385 · Cited by 12
Nie, Y., Wang, W., Zhou, F., Wang, T., Li, S., Liu, C., & Gao, J. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1624094 · Cited by 16
“This study used the Mixed Methods Appraisal Tool (MMAT 2018 version) to evaluate the methodological quality of the included quantitative observational studies (35).”
Uysal, D. A., & Ekiz, E. (2025) · BMC Nursing · doi:10.1186/s12912-025-04034-4 · Cited by 12
“The study is reported in accordance with the guidelines of the Mixed Methods Appraisal Tool [21].”
Single, N., Graham, W. M., Kelson, J. N., & Sulikowski, D. (2025) · JMIR Mental Health · doi:10.2196/71963 · Cited by 12
“Included studies were critically appraised with the Mixed Methods Appraisal Tool [48].”
Figueroa, R., Purkait, T., Saltzman, J., et al. (2025) · AJPM Focus · doi:10.1016/j.focus.2025.100444 · Cited by 7
“32 Mixed Methods Appraisal Tool is designed to critically appraise qualitative, quantitative, and mixed-methods studies that are included in evidence syntheses.”
Paes, V. M., Ting, A. S. J., Paes, M. V. I., Masters, J., Graham, S., & Costa, M. L. (2026) · The Bone & Joint Journal · doi:10.1302/0301-620x.108b1.bjj-2025-0035.r1 · Cited by 12
Digital Health Interventions to Promote Physical Activity Among Adolescents: Systematic Review
Fan, R. S., Jiang, J. J., Zhou, Q. Y., Zhang, X. Y., Wu, Z. H., & Ji, L. (2026) · Journal of Medical Internet Research · doi:10.2196/82395 · Cited by 7
Bryan, J., Ketley, A., Cavanagh, K., Bisset, C. N., Moug, S. J., Wyld, L., & Morgan, J. L. (2025) · British journal of surgery · doi:10.1093/bjs/znaf258 · Cited by 10
“A summary of the appropriate sections of the MMAT scores can be found in Table S1 .”
InfluentialHuman Factors Influencing Trust in Healthcare Artificial Intelligence: Systematic Literature Review
Choudhury, A., & Shamszare, H. (2026) · IISE Transactions on Occupational Ergonomics and Human Factors · doi:10.1080/24725838.2026.2625655 · Cited by 14
“The MMAT is designed to appraise the quality of empirical studies across diverse designs, including qualitative, quantitative descriptive, randomized controlled, non-randomized, and mixed-methods research.”
InfluentialCharacterising generalism in clinical practice: a systematic mixed studies review protocol
Kelly, M. A., Cheung, S., Keshavjee, M., Stevenson, A., Elliott, J., Singh, S., Foster, M., & Park, S. E. (2021) · BJGP Open · doi:10.3399/bjgpo.2021.0029 · Cited by 7
“Quality appraisal and data synthesis A wide range of critical appraisal tools have been developed in recent years to assess mixedmethods research, including systematic mixedmethods reviews.(18) Despite this, consensus on how best to evaluate study quality for mixedmethods studies is lacking, particularly in relation t…”
BackgroundPaterson, C. C., Owusu, E., & Rolley, J. X. (2021) · Seminars in Oncology Nursing · doi:10.1016/j.soncn.2021.151211 · Cited by 7
A Systematic Review of Interventions Used to Reduce University Students’ Speaking Anxiety
Elahemer, H. H. M., & Said, S. M. (2022) · International Journal of Education and Practice · doi:10.18488/61.v10i2.2994 · Cited by 20
“As depicted in Table 5, the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) was employed to evaluate the quality of mixed-method papers.”
MethodPorter, J. E., Dabkowski, E., Ghasemiardekani, M., Barbagallo, M. S., James, M. H., Prokopiv, V., & Wright, W. (2022) · Reviews on Environmental Health · doi:10.1515/reveh-2022-0061 · Cited by 9
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.