The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 201–300 of 4,578
Examining the impact of virtual reality on clinical decision making – An integrative review
Jans, C., Bogossian, F. E., Andersen, P., & Levett-Jones, T. L. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105767 · Cited by 94
Morisod, K., Malebranche, M. E., Marti, J., Spycher, J., Grazioli, V. S., & Bodenmann, P. (2022) · European Journal of Public Health · doi:10.1093/eurpub/ckac056 · Cited by 62
Peer mentoring programs for nursing students: A mixed methods systematic review
Lim, S., Lim, X. M., Chan, C. J. W., Dong, Y., Mikkonen, K., & Zhou, W. (2022) · Nurse Education Today · doi:10.1016/j.nedt.2022.105577 · Cited by 71
Rabies in Southeast Asia: a systematic review of its incidence, risk factors and mortality
Ling, M. Y. J., Halim, A. F. N. A., Ahmad, D., Ramly, N., Hassan, M. R., Rahim, S. S. S. A., Jeffree, M. S., Omar, A., & Hidrus, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-066587 · Cited by 91
Bear, H. A., Ayala‐Nunes, L., Ramos, G., Manchanda, T., Fernandes, B., Chabursky, S., Walper, S., Watkins, E. R., & Fazel, M. S. (2024) · Journal of Medical Internet Research · doi:10.2196/48964 · Cited by 78
“We used the Mixed Methods Appraisal Tool (MMAT; version 2018) to assess the methodological quality of the included studies [34].”
Treatment burden in multimorbidity: an integrative review
Lee, J. E., Lee, J., Shin, R., Oh, O., & Lee, K. S. (2024) · BMC Family Practice · doi:10.1186/s12875-024-02586-z · Cited by 93
“The ratings for the criteria were presented without calculating the overall score as recommended [30].”
Static, Dynamic, and Cognitive Fit of Exosystems for the Human Operator
Stirling, L., Kelty‐Stephen, D. G., Fineman, R. A., Jones, M. L. H., Park, B.-K. D., Reed, M. P., Parham, J. L., & Choi, H. J. (2020) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720819896898 · Cited by 73
“An evaluation of the references was performed following the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodGeddis‐Regan, A. R., Gray, D., Buckingham, S., Misra, U., Boyle, C., & the British Society for Disability and Oral Health (2022) · Special Care in Dentistry · doi:10.1111/scd.12652 · Cited by 65
Ackerhans, S., Huynh, T. V., Kaiser, C., & Schultz, C. (2024) · Implementation Science · doi:10.1186/s13012-024-01339-x · Cited by 85
“The three authors independently assessed the methodological quality of these 220 selected studies using the Mixed Methods Appraisal tool (MMAT), version 2018 [58].”
BackgroundMethodApplication of artificial intelligence in triage in emergencies and disasters: a systematic review
Tahernejad, A., Sahebi, A., Abadi, A. S. S., & Safari, M. (2024) · BMC Public Health · doi:10.1186/s12889-024-20447-3 · Cited by 89
Eisner, E., Faulkner, S., Allan, S., Ball, H., Basilio, D. D., Nicholas, J., Priyam, A., Wilson, P. M., Zhang, X., & Bucci, S. (2025) · JMIR Mental Health · doi:10.2196/65246 · Cited by 77
Robotics in the Construction Sector: Trends, Advances, and Challenges
Liu, Y., A.H., A., Haron, N. A., N.A., B., & Wang, H. (2024) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-024-02104-4 · Cited by 60
“Since the review focused on mixed research designs (quantitative + qualitative + mixed methods), the quality of the selected publications was assessed using the Mixed Methods Appraisal Tool (MMAT) version 2018 [18].”
Delivering brief physical activity interventions in primary care: a systematic review
Hall, L. H., Thorneloe, R., Rodriguez-Lopez, R., Grice, A., Thorat, M. A., Bradbury, K., Kamble, M. W., Okoli, G. N., Powell, D., & Beeken, R. J. (2022) · British Journal of General Practice · doi:10.3399/bjgp.2021.0312 · Cited by 59
Innovation in dementia education within undergraduate healthcare programmes: A scoping review
Williams, M., & Daley, S. (2020) · Nurse Education Today · doi:10.1016/j.nedt.2020.104742 · Cited by 65
Digital mental health in China: a systematic review
Zhang, X., Lewis, S. W., Firth, J., Xu, C., & Bucci, S. (2021) · Psychological Medicine · doi:10.1017/s0033291721003731 · Cited by 71
“Due to the diversity of study designs included, the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) was used to assess the quality of studies.”
BackgroundMethodInfluentialVisit Types in Primary Care With Telehealth Use During the COVID-19 Pandemic: Systematic Review
Ward, K., Vagholkar, S., Sakur, F., Khatri, N. N., & Lau, A. Y. S. (2022) · JMIR Medical Informatics · doi:10.2196/40469 · Cited by 68
Impact of Social Determinants of Health on Hypertension Outcomes: A Systematic Review
Metlock, F., Hinneh, T. K., Benjasirisan, C., Alharthi, A., Ogungbe, O., Turkson‐Ocran, R., Himmelfarb, C. D., & Commodore-Mensah, Y. Y. (2024) · Hypertension · doi:10.1161/hypertensionaha.123.22571 · Cited by 86
“77 • Quantitative (nonrandomized) studies: representativeness of the sample to the target population, use of appropriate measurements, completeness of outcome data, accountability of confounders, and intervention administered as intended.”
Pietraszewski, P., Terbalyan, A., Roczniok, R., Maszczyk, A., Ornowski, K., Manilewska, D., Kuliś, S., Zając, A., & Gołaś, A. (2025) · Applied Sciences · doi:10.3390/app15137254 · Cited by 119
A systematic review of religion/spirituality and sport: A psychological perspective
Noh, Y., & Shahdan, S. (2019) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2019.101603 · Cited by 82
Campbell, L., Masquillier, C., Thunnissen, E., Ariyo, E., Tabana, H., Sematlane, N. P., Delport, A., Dube, L. T., Knight, L., Flechner, T. K., et al. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17113808 · Cited by 81
“The quantitative studies were evaluated using the National Institutes of Health (NIH) Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies systematic review [53], and the mixed methods by using the Mixed Methods Appraisal Tool (MMAT) [54].”
MethodA Systematic Review on Barriers to Palliative Care in Oncology
Parajuli, J., & Hupcey, J. E. (2021) · American Journal of Hospice and Palliative Medicine® · doi:10.1177/1049909120983283 · Cited by 61
“Since the calculation of an overall score from the ratings of each criterion is discouraged by current literature, the overall score was removed from the current version of the MMAT tool.(24) The tool now has a more detailed description of the ratings for each criterion to better inform the quality of the included stu…”
MethodInfluentialNurse Managers’ Leadership, Patient Safety, and Quality of Care: A Systematic Review
Lee, S. E., Lee, H., & Sang, S. (2022) · Western Journal of Nursing Research · doi:10.1177/01939459221114079 · Cited by 87
“The Mixed-Method Appraisal Tool (MMAT), which has been found to be an efficient and reliable tool (Hong et al., 2018), was used to evaluate the methodological quality of the 14 selected studies.”
MethodInfluentialThe development of community paramedicine; a restricted review
Shannon, B., Eaton, G., Lanos, C., Leyenaar, M. S., Nolan, M., Bowles, K., Williams, B. A., O'Meara, P. F., Wingrove, G., Heffern, J., et al. (2022) · Health & Social Care in the Community · doi:10.1111/hsc.13985 · Cited by 83
“A risk- of- bias appraisal of included peer- reviewed literature using the Mixed Methods Appraisal Tool (MMAT) was conducted (Hong et al., 2018).”
MethodBai, X., Chen, M., He, R., & Xu, T. (2022) · Journal of Family Theory & Review · doi:10.1111/jftr.12478 · Cited by 73
The Mental Health of Elite-Level Coaches: A Systematic Scoping Review
Frost, J., Walton, C. C., Purcell, R., Fisher, K., Gwyther, K., Kocherginsky, M., & Rice, S. M. (2024) · Sports Medicine - Open · doi:10.1186/s40798-023-00655-8 · Cited by 59
“As a result, this review employed Hong et al.’s [75] Mixed Methods Appraisal Tool (MMAT), as the instrument provides reviewers with a single tool to assess quantitative, qualitative and mixed methods research, and has shown improved content validity over recent years [76].”
BackgroundMethodDarley, S., Coulson, T., Peek, N., Moschogianis, S., van der Veer, S. N., Wong, D. C., & Brown, B. C. (2022) · Journal of Medical Internet Research · doi:10.2196/37436 · Cited by 53
“The methodological quality of the studies was assessed using the Mixed Methods Appraisal Tool (MMAT), which is designed for qualitative, quantitative, and mixed methods studies [76].”
Qudah, B. A., Thakur, T., & Chewning, B. (2021) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2021.03.005 · Cited by 50
General practice pharmacists in Australia: A systematic review
Sudeshika, T., Naunton, M., Deeks, L. S., Thomas, J., Peterson, G. M., & Kosari, S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258674 · Cited by 56
“The MMAT includes five core quality criteria for each of the following five categories of study designs: qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies, and mixed-method studies [68].”
MethodSystematic review of pathways to care in the U.S. for Black individuals with early psychosis
Oluwoye, O. A., Davis, B. J., Kuhney, F. S., & Anglin, D. M. (2021) · Schizophrenia · doi:10.1038/s41537-021-00185-w · Cited by 65
Diversity in public perceptions of urban forests and urban trees: A critical review
Ordóñez, C., Wolf, K. L., Kowalski, J. M., Kendal, D., Byrne, J., & Conway, T. M. (2022) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2022.104466 · Cited by 80
Sleep disturbances in Lewy body dementia: A systematic review
Elder, G. J., Lázár, A. S., Alfonso‐Miller, P., & Taylor, J. (2022) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5814 · Cited by 66
Phang, J. K., Kwan, Y. H., Yoon, S., Goh, H., Yee, W. Q., Tan, C. S., & Low, L. L. (2022) · JMIR Aging · doi:10.2196/39848 · Cited by 81
“The methodological quality of evidence (rigor) around each therapy was assessed using the Mixed Methods Appraisal Tool (MMAT) [50].”
MethodInfluentialLynch, L., Moorhead, A., Long, M., & Hawthorne-Steele, I. (2023) · Journal of Child and Family Studies · doi:10.1007/s10826-022-02498-5 · Cited by 58
“Overall, the studies included were predominantly high quality which in part might be explained by the intensive screening processes employed, which eliminated articles early on with questionable methodology (Hong et al. 2018).”
BackgroundMethodInfluentialSpeaking of the devil: a systematic literature review on community preparedness for earthquakes
Shaffril, H. A. M., Samah, A. A., & Kamarudin, S. (2021) · Natural Hazards · doi:10.1007/s11069-021-04797-4 · Cited by 93
“MMAT enables researchers to appraise a systematic mixed studies review and covers the appraisal of five kinds of studies: qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies, and mixed methods studies (Hong et al. 2018).”
BackgroundHopstaken, J. S., Verweij, L., van Laarhoven, C. J. H. M., Blijlevens, N. M. A., Stommel, M. W. J., & Hermens, R. (2021) · Journal of Medical Internet Research · doi:10.2196/28869 · Cited by 67
“The Mixed Methods Appraisal Tool (MMAT) version 2018 was used to critically appraise the included studies [36].”
Myloneros, T., & Sakellariou, D. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06678-9 · Cited by 72
“The Mixed Methods Appraisal Tool (MMAT) [26] was used for published literature and the AACODS [27] checklist for grey literature (see supplementary materials C, D, E, and F).”
MethodPawils, S., Heumann-Schoop, S., Schneider, S. A., Metzner, F., & Mays, D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0275724 · Cited by 72
Wu, Y., Kemps, E., & Prichard, I. (2024) · Body Image · doi:10.1016/j.bodyim.2023.101650 · Cited by 60
“Finally, the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) was used to assess the quality of mixed methods and qualitative studies.”
BackgroundMethodInfluentialPreventive Interventions for Internet Addiction in Young Children: Systematic Review
Theopilus, Y., Mahmud, A. A., Davis, H., & Octavia, J. R. (2024) · JMIR Mental Health · doi:10.2196/56896 · Cited by 94
“To assess the risk of bias, we evaluated each study using the Mixed Methods Appraisal Tool (MMAT; version 2018; McGill University) [49].”
MethodA systematic review of the effects of chronic, slow-onset climate change on mental health
Burrows, K., Denckla, C. A., Hahn, J., Schiff, J. E., Okuzono, S. S., Randriamady, H., Mita, C., Kubzansky, L. D., Koenen, K. C., & Lowe, S. R. (2024) · Nature Mental Health · doi:10.1038/s44220-023-00170-5 · Cited by 58
“To evaluate the extent to which the included studies were protected against bias, we used modified versions of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool 41 for quantitative studies, the Mixed Methods Appraisal Tool 42 for mixed-methods studies…”
Health Care Professionals’ Experience of Using AI: Systematic Review With Narrative Synthesis
Ayorinde, A., Mensah, D. O., Walsh, J., Ghosh, I., Ibrahim, S. A., Hogg, J., Peek, N., & Griffiths, F. (2024) · Journal of Medical Internet Research · doi:10.2196/55766 · Cited by 55
Self-citationPiat, M., Wainwright, M., Sofouli, E., Vachon, B., Deslauriers, T., Préfontaine, C., & Frati, F. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01646-0 · Cited by 73
“Results of study quality assessment and sensitivity analysis Of the 70 included studies, 55 were categorized as qualitative, six mixed methods, six quantitative descriptive, one quantitative non-RCT, one quantitative RCT, and one as both quantitative descriptive and qualitative for the purposes of the MMAT assessment.”
MethodInfluentialMeherali, S. M., Rahim, K. A., Campbell, S., & Lassi, Z. S. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.761394 · Cited by 108
“We assessed the methodological quality of the quantitative and qualitative papers using the Mixed-Method Appraisal Tool (M-MAT) (25).”
MethodChoy, C. L., Liaw, S. Y., Goh, E. L., See, K. C., & Chua, W. L. (2022) · Journal of Hospital Infection · doi:10.1016/j.jhin.2022.01.004 · Cited by 106
Virtual Reality Applications in Medicine During the COVID-19 Pandemic: Systematic Review
Pallavicini, F., Pepe, A., Clerici, M., & Mantovani, F. (2022) · JMIR Serious Games · doi:10.2196/35000 · Cited by 75
Moonen, C. P. B., den Heijer, C. D. J., Dukers–Muijrers, N. H. T. M., van Dreumel, R., Steins, S. C. J., & Hoebe, C. J. P. A. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1118227 · Cited by 60
“Quality Assessment MMAT.”
MethodOverbury, K., Conroy, B. W., & Marks, E. M. (2023) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2023.102073 · Cited by 77
Sachs, A. L., Kolster, A., Wrigley, J., Papon, V., Opačin, N., Hill, N., Howarth, M. L., Rochau, U., Hidalgo, L., Casajuana, C., et al. (2024) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2024.105071 · Cited by 78
Exploring Problematic TikTok Use and Mental Health Issues: A Systematic Review of Empirical Studies
Jain, L., Vélez, L. A., Karlapati, S. K., Forand, M., Kannali, R., Yousaf, R. A., Ahmed, R., Sarfraz, Z., Sutter, P. A., Tallo, C., et al. (2025) · Journal of Primary Care & Community Health · doi:10.1177/21501319251327303 · Cited by 124
Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G. (2024) · Journal of Medical Internet Research · doi:10.2196/55897 · Cited by 62
“Critical appraisal of the studies selected for data synthesis was performed using the Mixed Method Appraisal Tool (MMAT [29]), which has been designed specifically for systematic reviews that include heterogeneous studies, as it allows to assess the methodological quality of 5 types of study…”
Cook, N., Collins, J., Goodwin, D. M., & Porter, J. A. (2021) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.12928 · Cited by 66
Juhrmann, M. L., Vandersman, P., Butow, P. N., & Clayton, J. M. (2021) · Palliative Medicine · doi:10.1177/02692163211059342 · Cited by 76
Hoskote, A. R., Croce, E. A., & Johnson, K. E. (2022) · The Journal of School Nursing · doi:10.1177/10598405211068120 · Cited by 84
“(Hong et al., 2018, p. 2).”
MethodYusop, S. R. M., Rasul, M. S., Mohamad Yasin, R., Hashim, H. U., & Jalaludin, N. A. (2022) · Sustainability · doi:10.3390/su14095225 · Cited by 57
“the Mixed Methods Appraisal Tool (MMAT) version 2018 [27].”
BackgroundMethodWu, S., Tannous, E. E., Haldane, V. E., Ellen, M., & Wei, X. (2022) · Implementation Science · doi:10.1186/s13012-022-01209-4 · Cited by 73
“Quality appraisal Since we expected to include studies with a wide range of methodologies, the quality of included studies was assessed using the Mixed Methods Appraisal Tool (Version 2018), which appraises the methodological quality of five categories of studies: qualitative research, randomized controlled trials, no…”
MethodJames, H., Morgan, J., Ti, L., & Nolan, S. (2023) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2023.109763 · Cited by 66
Kúld, P. B., Frielink, N., Zijlmans, M., Schuengel, C., & Embregts, P. J. C. M. (2023) · Journal of Intellectual Disability Research · doi:10.1111/jir.13036 · Cited by 73
Naya, K., Aikawa, G., Ouchi, A., Ikeda, M., Fukushima, A., Yamada, S., Kamogawa, M., Yoshihara, S., & Sakuramoto, H. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292108 · Cited by 67
“Instead of generating an overall score for each study, a qualitative approach was applied by providing a detailed review of study quality [11].”
MethodFactors Impacting Retention of Aged Care Workers: A Systematic Review
Thwaites, C. K., McKercher, J. P., Fetherstonhaugh, D., Blackberry, I., Gilmartin‐Thomas, J., Taylor, N. F., Bourke, S. L., Fowler‐Davis, S., Hammond, S., & Morris, M. E. (2023) · Healthcare · doi:10.3390/healthcare11233008 · Cited by 74
“The mixed-methods appraisal tool (MMAT) [39] was used by two authors independently (C.T., J.M.) to assess methodological quality of the selected studies.”
MethodTechnology integration in complex healthcare environments: A systematic literature review
Bayramzadeh, S., & Aghaei, P. (2021) · Applied Ergonomics · doi:10.1016/j.apergo.2020.103351 · Cited by 71
Hindmarch, S., Hawkes, R. E., Howell, S. J., Gorman, L. S., & French, D. (2021) · Cancers · doi:10.3390/cancers13164150 · Cited by 62
“Quality Assessment The Mixed Methods Appraisal Tool (MMAT) was deemed most suitable for quality assessment due to its demonstrated reliability and inclusion of quality criteria specifically designed to appraise quantitative descriptive study designs such as surveys [30].”
MethodPhysiological and cognitive demands of orienteering: a systematic review
Batista, M. M., Paludo, A. C., Gula, J. N., Pauli, P. H., & Tartaruga, M. P. (2020) · Sport Sciences for Health · doi:10.1007/s11332-020-00650-6 · Cited by 54
Context, Input, Process, and Product Evaluation Model in medical education: A systematic review
Toosi, M., Modarres, M., Amini, M., & Geranmayeh, M. (2021) · Journal of Education and Health Promotion · doi:10.4103/jehp.jehp_1115_20 · Cited by 55
“Each study was given an overall quality score (unclassified, 25%, 50%, 75%, or 100%) based on the MMAT scoring system.[12,13] Results”
MethodAeen, F. B., Pakzad, R., Rad, M. G., Abdi, F., Zaheri, F., & Mirzadeh, N. (2021) · Scientific Reports · doi:10.1038/s41598-021-93739-y · Cited by 58
Nursing education and training on electronic health record systems: An integrative review
Ting, J. J., Garnett, A., & Donelle, L. (2021) · Nurse Education in Practice · doi:10.1016/j.nepr.2021.103168 · Cited by 53
Youth sport dropout according to the Process-Person-Context-Time model: a systematic review
Moulds, K., Galloway, S., Abbott, S., & Cobley, S. P. (2022) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.2012817 · Cited by 74
“…fi ve methodological criteria for fi ve di ff erent types of study designs (qualitative studies, 3 randomised controlled trials, non-randomised quantitative studies, quantitative descriptive and mixed methods studies) rated on a nominal scale ( ‘ yes ’ , ‘ no ’ , ‘ can ’ t tell ’ ; Hong et al., 2018).”
MethodInfluentialEducational interventions to promote respectful maternity care: A mixed-methods systematic review
Dhakal, P., Creedy, D. K., Gamble, J. A., Newnham, E. C., & McInnes, R. J. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103317 · Cited by 73
Goodall, G., Mjøen, O. M., Witsø, A. E., Horghagen, S., & Kvam, L. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.882066 · Cited by 81
“The Critical Appraisal Skills Programme [CASP] (2018) checklist for qualitative research was used to assess qualitative studies and the Mixed Method Appraisal Tool (MMAT) version 2018 (Hong et al., 2018) was used to assess quantitative and mixed methods studies.”
MethodSedani, A. E., Davis, O. C., Clifton, S. C., Campbell, J. E., & Chou, A. F. (2022) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djac154 · Cited by 66
“The manual discour-ages the calculation of an overall score from the ratings of each criterion and instead advises individuals to provide a more detailed narrative explaining the ratings for each criterion to better inform the quality of included studies (20).”
Pettersen, S., Eide, H., & Berg, A. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-10867-7 · Cited by 73
“The MMAT [26] was employed to assess the quality of the 23 included studies.”
MethodA Systematic Review of User Expectations and Outcomes of Crisis Support Services
Mazzer, K., O’Riordan, M., Woodward, A., & Rickwood, D. (2020) · Crisis · doi:10.1027/0227-5910/a000745 · Cited by 55
Nkambule, S. J., Moodley, I., Kuupiel, D., & Mashamba-Thompson, T. P. (2021) · Scientific Reports · doi:10.1038/s41598-021-84344-0 · Cited by 57
de Jong, Y., van der Willik, E. M., Milders, J., Voorend, C. G. N., Morton, R. L., Dekker, F. W., Meuleman, Y., & van Diepen, M. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01363-1 · Cited by 70
“checklists, both designed for reporting- or overall quality assessment, such as the CASP [30], QualSyst [31], GRADE-CREQual [32], MMAT [33], amongst others.”
BackgroundFatema, S. R., East, L., Islam, M. S., & Usher, K. J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182111068 · Cited by 62
“The mixed-method study was assessed as medium quality article as per the MMAT.”
MethodInfluentialJones, C., Jones, D., & Moro, C. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053616 · Cited by 61
Foroughi, Z., Ebrahimi, P., Aryankhesal, A., Maleki, M., & Yazdani, S. (2022) · BMC Public Health · doi:10.1186/s12889-022-12496-3 · Cited by 71
“This tool is applicable for quality assessment of systematic mixed studies review that needs to quality assessment of various quantitative, qualitative, and mixed-method studies [13].”
MethodTaylor, L. C., Hutchinson, A. M., Law, K., Shah, V., Usher‐Smith, J. A., & Dennison, R. A. (2023) · Health Expectations · doi:10.1111/hex.13739 · Cited by 58
The association between frailty and anxiety: A systematic review
Tan, M., Bhanu, C., & Frost, R. (2023) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5918 · Cited by 56
“…(MT, RF) assessed study quality of the eligible studies using the Mixed Methods Appraisal tool (MMAT) for quantitative non ‐ randomised studies, 16 which assesses five key domains regarding representativeness, measures used, missing data, confounders and whether the exposure occurred as…”
MethodCareer progression for autistic people: A scoping review
Davies, J., Romualdez, A. M., Pellicano, E., & Remington, A. (2024) · Autism · doi:10.1177/13623613241236110 · Cited by 52
“We assessed the quality of eligible studies using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
MethodPearson, J., Giacumo, L. A., Farid, A., & Sadegh, M. (2022) · Education Sciences · doi:10.3390/educsci12050333 · Cited by 51
Hassan, S. Z., Chan, V. W. S., Stevens, J. E., & Stupans, I. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01577-w · Cited by 58
“Quality assessment using the Mixed Methods Appraisal Tool (MMAT) Version 2018.”
MethodInfluentialSaab, M. M., Fitzgerald, S. M., Noonan, B. J., Kilty, C., Collins, A. R., Lyng, Á., Kennedy, U. M., O’Brien, M., & Hegarty, J. M. (2021) · Health Promotion International · doi:10.1093/heapro/daab016 · Cited by 50
“The Mixed Methods Appraisal Tool (MMAT) helps appraise the methodological quality of five study categories: qualitative studies, randomized controlled trials (RCTs), nonrandomized studies, quantitative descriptive studies and mixed methods studies (Hong et al., 2018).”
MethodAnalysing interventions designed to reduce tuberculosis-related stigma: A scoping review
Foster, I., Galloway, M., Human, W., Anthony, M. G., Myburgh, H., Vanqa, N., Wademan, D. T., Makanda, G., Tisile, P., Schoeman, I., et al. (2022) · PLOS Global Public Health · doi:10.1371/journal.pgph.0000989 · Cited by 63
Bear, H. A., Ayala Nunes, L., DeJesus, J., Liverpool, S., Moltrecht, B., Neelakantan, L., Harriss, E., Watkins, E., & Fazel, M. (2022) · Journal of Medical Internet Research · doi:10.2196/40347 · Cited by 51
“The mixed methods appraisal tool (version 2018) was used to assess the methodological quality of the included studies [33].”
MethodGemmell, E., Ramsden, R., BRUSSONI, M. J., & Brauer, M. D. (2022) · Journal of Urban Health · doi:10.1007/s11524-022-00696-6 · Cited by 76
“Two authors independently assessed the quality of included studies using the Mixed Methods Appraisal Tool (MMAT) [27] resolving differences through discussion.”
MethodInfluentialKomanchuk, J., Toews, A. J., Marshall, S., MacKay, L. J., Hayden, A., Cameron, J. L., Duffett‐Leger, L., & Letourneau, N. L. (2023) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2022.0278 · Cited by 86
“To promote consistency, we used the Mixed Methods Appraisal Tool (MMAT), which was developed for mixed (e.g., qualitative, mixed-methods) studies, 25 instead of utilizing multiple tools as outlined in our protocol.”
Moral sensitivity of nursing students: a systematic review
Shadi, A. Z., Vanaki, Z., Eesa, M., & Anoshirvan, K. (2024) · BMC Nursing · doi:10.1186/s12912-024-01713-6 · Cited by 75
“The methodological quality of the included studies was assessed independently by two authors using the Mixed Methods Appraisal Tool (MMAT) [20] and PRISMA-ScR [21].”
MethodHealth challenges faced by parents of children with disabilities: a scoping review
Shahali, S., Tavousi, M., Sadighi, J., Kermani, R. M., & Rostami, R. (2024) · BMC Pediatrics · doi:10.1186/s12887-024-05104-3 · Cited by 97
“The Mixed Methods Appraisal Tool (MMAT) was used for the quality assessment of the articles [19].”
MethodTransitional Care Interventions for Patients with Heart Failure: An Integrative Review
Ba, H. M., Son, Y., Lee, K., & Kim, B.-H. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17082925 · Cited by 67
“The quality of studies with different designs was assessed using the Mixed Methods Appraisal Tool (MMAT; 2018 version), which is a critical tool designed for the appraisal stage of systematic mixed methods studies, including integrative reviews [15].”
MethodEffect of community-based education on undergraduate nursing students’ skills: a systematic review
Zeydani, A., Atashzadeh‐Shoorideh, F., Abdi, F., Hosseini, M., Anboohi, S. Z., & Skerrett, V. (2021) · BMC Nursing · doi:10.1186/s12912-021-00755-4 · Cited by 80
“The methodological quality of the included studies assessed independently by two authors using the Mixed Methods Appraisal Tool (MMAT) [17].”
MethodInfluentialA systematic review of working conditions and occupational health in home office
Wütschert, M. S., Pereira, D., Suter, L., Schulze, H., & Elfering, A. (2022) · Work · doi:10.3233/wor-205239 · Cited by 102
“The quality assessment was evaluated using the mixed methods appraisal tool (MMAT) from Hong, Pluye [39].”
BackgroundMethodMatch officials and abuse: a systematic review
Mojtahedi, D., Webb, T., Leadley, C. B., & Jones, M. (2022) · European Sport Management Quarterly · doi:10.1080/16184742.2022.2099441 · Cited by 65
“Quality appraisals for each study were assessed by two investigators using the Mixed Methods Appraisal Tool (MMAT), version 2018 (Appendix C; Hong et al., 2018).”
MethodDakin, C. A., Beaulieu, K., Hopkins, M., Gibbons, C., Finlayson, G., & Stubbs, R. J. (2022) · Obesity Reviews · doi:10.1111/obr.13515 · Cited by 68
“The MMAT has been pilot tested and has high content validity.(38) The following questions were answered to determine the risk of bias: is the sampling strategy relevant to address the research question, is the sample representative of the target population, are the measurements appropriate, is the risk of nonresponse…”
Bdaiwi, Y., Sabouni, A., Patel, P., Ekzayez, A., Alchalati, S., Abdrabbuh, O., Abbara, A., & Głogowska, M. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-064851 · Cited by 51
Jouanny, C., Abhyankar, P., & Maxwell, M. (2024) · BMC Women s Health · doi:10.1186/s12905-024-03063-6 · Cited by 80
“The Mixed Methods Appraisal Tool (MMAT) [35] was used to appraise the methodological quality of each study by the author, and jointly for 33% of studies by a research assistant.”
MethodAn integrative literature review of the CSR-HRM nexus: Learning from research-practice gaps
Podgorodnichenko, N., Edgar, F., & Akmal, A. (2021) · Human Resource Management Review · doi:10.1016/j.hrmr.2021.100839 · Cited by 76
Integrating nursing informatics into undergraduate nursing education in Africa: A scoping review
Harerimana, A., Wicking, K., Biedermann, N., & Yates, K. (2020) · International Nursing Review · doi:10.1111/inr.12618 · Cited by 62
“The selection of articles was mainly based on the research question and minimum standards of quality were considered based on the ‘Mixed Methods Appraisal Tool (MMAT)’ (Hong et al. 2018, p. 2).”
MethodBoland, J. W., Brown, M. E. L., Dueñas, A. N., Finn, G. M., & Gibbins, J. L. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-036458 · Cited by 48
Effects of mHealth‐based interventions on health literacy and related factors: A systematic review
Lin, Y.-H. P., & Lou, M. (2020) · Journal of Nursing Management · doi:10.1111/jonm.13175 · Cited by 63
Stuart, G., & D’Lima, D. (2021) · Psychology and Health · doi:10.1080/08870446.2021.1918690 · Cited by 53
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.