The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,501–1,600 of 4,578
Hunter, J. D., Hetherington, K., Courtney, E., Christensen, Y., Fuentes-Bolanos, N., Bhatia, K., & Peate, M. (2024) · Genetics in Medicine · doi:10.1016/j.gim.2024.101197 · Cited by 15
Hemifacial microsomia: a scoping review on progressive facial asymmetry due to mandibular deformity
Atiba, P. M., Omotoso, B. R., Madaree, A., et al. (2024) · Oral and Maxillofacial Surgery · doi:10.1007/s10006-024-01276-5 · Cited by 10
“Quantitative descriptive criteria of the Mixed Methods Appraisal Tool (MMAT)-Version 2018 were used for the included studies [34].”
Stewart, A., Govender, R., Eaton, S. D., Smith, C. H., Coppi, P. D., & Wray, J. (2024) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-024-03259-x · Cited by 11
Smartphone-Based Hand Function Assessment: Systematic Review
Fu, Y. H., Zhang, Y., Ye, B., Babineau, J., Zhao, Y., Gao, Z., & Mihailidis, A. (2024) · Journal of Medical Internet Research · doi:10.2196/51564 · Cited by 12
“After the screening stage, the research quality of selected studies was evaluated using the Mixed Methods Appraisal Tool, a tool designed for the systematic mixed research review evaluation phase [22].”
Sleep and socio-occupational functioning in adults with serious mental illness: A systematic review
Stafford, A., Oduola, S., & Reeve, S. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116111 · Cited by 16
“The methodological quality of the studies was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Acero, Á. E., Lobato, J. M. P., & Sumaza, C. R. (2024) · International Social Work · doi:10.1177/00208728241269670 · Cited by 16
“Following the directions of the 2018 version of the MMAT (Hong et al., 2018), two authors (AE and JMP) independently rated each of the criteria included in the tool, using ‘yes’, ‘no’ or ‘can’t tell’ responses and incorporating qualitative comments for each.”
Voice assistants’ influence on loneliness in older adults: a systematic review
Martínez, E. C., Encuentra, E. H., & Pousada, M. (2024) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2024.2397030 · Cited by 17
“the methodological quality of the included studies was assessed with the Mixed Methods Appraisal tool (MMAt) [31].”
Loneliness in sport: A systematic mixed studies review
Jackman, P. C., Hawkins, R. M., Bird, M. D., Williamson, O., Vella, S. A., & Lazuras, L. (2024) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2024.2405507 · Cited by 14
“To appraise the quality of primary studies, we used the mixed methods appraisal tool (MMAT; Hong et al., 2018).”
Barriers and Facilitators to Cycling to School for Children in the UK: A Systematic Review
Boland, P., Nowland, R. A., Tellis, K. D., Adams, M. D., Westwood, J., Crook, D. J., Larkins, C., & Ridley, J. (2024) · Active Travel Studies · doi:10.16997/ats.1553 · Cited by 11
Systematic review of what people know about brain health
Niechcial, M. A., Elhag, S., Potter, L. M., Dickson, A., & Gow, A. J. (2024) · Ageing Research Reviews · doi:10.1016/j.arr.2024.102592 · Cited by 10
Smith, C. D., Morgan, E. R., & Jones, R. A. (2024) · Parasitology · doi:10.1017/s0031182024000957 · Cited by 17
“After screening, risk of bias was assessed using the MMAT (Mixed Methods Appraisal Tool) (Nha Hong et al ., 2018).”
Figueroa, R., Owoputi, I., Liu, R., Chiong, R., Salas, J. F., Saltzman, J. A., & Baker, K. (2024) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/23727322241304104 · Cited by 13
“MMAT is designed to critically appraise qualitative, quantitative, and mixed methods studies that are included in systematic reviews.”
Interdisciplinary cross-sectoral strategies to mitigate health workforce migration in Africa
Oke, G. I. (2025) · Journal of Medicine Surgery and Public Health · doi:10.1016/j.glmedi.2025.100179 · Cited by 12
Gender-based violence syndemics in global health: A systematic review
Vahedi, L., Seff, I., Tsai, A. C., Rfat, M., Aljamhan, M. S., & Stark, L. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.117793 · Cited by 14
The relationship between housing and asylum seekers’ mental health: A systematic review
Spira, J., Katsampa, D., Wright, H., & Komolafe, K. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.117814 · Cited by 14
“The MMAT was utilised by the first author (JS) to assess study characteristics, such as sample representativeness, appropriateness of measurement used, and completeness of outcome data.”
Rathnayake, D., Sasame, A., Radomska, A., et al. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12388-3 · Cited by 10
Carlon, H. A., Hurlocker, M. C., Hoeppner, B. B., & Witkiewitz, K. A. (2025) · Addiction · doi:10.1111/add.70019 · Cited by 17
McGuire, C. M., Boskovic, N., Fatusin, B. B., Ameh, P. O., Reed, T., Jethwani, P., Flynn, D., Cooke, J., & Saper, R. B. (2025) · Annals of Global Health · doi:10.5334/aogh.4543 · Cited by 13
“To understand the methodological rigor of the studies in our sample, we used the Mixed Methods Appraisal Tool (MMAT) to assess program evaluations described in eligible empirical studies [41].”
A systematic review of sport-based adolescent mental health awareness programmes
Sullivan, N., Breslin, G., McLaughlin, M., Shannon, S., Leavey, G., & Dempster, M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315315 · Cited by 13
Koppelman, L. J. M., Oyugi, A. A., Maljaars, J., & Jong, A. E. V. D. M. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14072296 · Cited by 14
“…were categorized based on the main predictors or risk factors they studied, which were grouped into six overarching themes: lifestyle factors, non-IBD medications, psychological and behavioral factors, Coronavirus Disease 2019 (COVID-19), environmental factors, and multifactorial assessments.”
Evidence for Building an Integrated Dementia Healthcare System: A Systematic Review
Lin, G., Lin, T. K., Alluhidan, M., Saber, R., Hamza, M. M., Rakić, S., Alfaisal, A., Alamri, A., Çetinkaya, V., & Al-Azmi, N. (2025) · Clinical Gerontologist · doi:10.1080/07317115.2025.2505577 · Cited by 12
“The review team assessed the risk of bias using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
Measuring Parental School Involvement: A Systematic Review
Mocho, H., Martins, C. S., Santos, R., Ratinho, E., & Nunes, C. (2025) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe15060096 · Cited by 39
“These scores range from moderate to excellent, and no studies were excluded based on the MMAT criteria (Appendix B; Table A2).”
InfluentialMuralidhar, M., Delray, S., Cooper, C., Zabihi, S., Banerjee, S., Giebel, C. M., Dening, K. H., Birks, Y. F., Kenten, C., & Walpert, M. (2025) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.70119 · Cited by 10
A comprehensive review of traditional irrigation systems: Sustainability and future prospects
da-Silva-Branco, C., de Brito, A. G., & Seixas, P. C. (2025) · Agricultural Systems · doi:10.1016/j.agsy.2025.104481 · Cited by 18
Rouncivell, L., Takuva, S. G., Ledibane, N. R. T., Musekiwa, A., & Leong, T. D. D. (2021) · Tropical Medicine & International Health · doi:10.1111/tmi.13586 · Cited by 10
“These studies were subsequently assessed using the Mixed Methods Appraisal Tool (MMAT) version 18 [32].”
MethodPower, M., Arafa, N., Wenz, A., & Foley, G. (2021) · International Journal of Therapy and Rehabilitation · doi:10.12968/ijtr.2020.0144 · Cited by 14
“The MMAT comprises a five-item checklist for overall appraisal of a mixed-methods study - in addition to a five-item checklist that appraises the qualitative component of a mixed-methods study and a five-item checklist as appropriate to appraise the quantitative component of a mixed-methods study.”
MethodInfluentialBadu, E., O’Brien, A. P., & Mitchell, R. J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168873 · Cited by 12
“The authors developed a critical appraisal checklist using the Mixed-Methods Appraisal Tool (MMAT) [20] and the Joanna Briggs Institute (JBI) [21] critical appraisal tool.”
MethodCurrent methods of nurse-surgeon training and education: Systematic review
Grota, T., Betihavas, V., Burston, A., et al. (2021) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2021.100048 · Cited by 10
“Despite critical appraisal being undertaken, we acknowledge that the included studies did not score high for methodological rigour and quality using the Mixed Methods Appraisal Tool (Hong et al., 2018).”
BackgroundMethodInfluentialNjue, C., Sharmin, S., & Dawson, A. J. (2021) · Midwifery · doi:10.1016/j.midw.2021.103187 · Cited by 13
“The quality of the nine articles was assessed using validated tools ( CASP, 2018 ; Hong et al., 2018 ) and each article given an overall rating as ‘strong’ (70% score),’ moderate’ (55% score), or ‘weak’ (40% score).”
MethodCarandang, R. R., Sakamoto, J. L., Kunieda, M. K., Shibanuma, A., Львовна, Я. Е., Александровна, Б. М., & Jimba, M. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058155 · Cited by 11
Life Review Intervention: A Systematic Review of Its Feasibility Among Persons With Cancer
Ng, J. K.-Q., Subramaniam, P., Ismail, F., Ahmad, M., & Shamsudin, N. L. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2022.07.016 · Cited by 14
One for all or all for one? An integrative review of research on frequent callers
Skogevall, S., Kaminsky, E., Håkansson, J., & Holmström, I. K. (2022) · PEC Innovation · doi:10.1016/j.pecinn.2022.100070 · Cited by 10
“The remaining articles were assessed according to the Mixed Methods Appraisal Tool for critical appraisal [13].”
MethodA last resort? A scoping review of patient and healthcare worker attitudes toward strike action
Essex, R., Burns, C., Evans, T. R., Hudson, G., Parsons, A., & Weldon, S. M. (2022) · Nursing Inquiry · doi:10.1111/nin.12535 · Cited by 13
“Each scale includes five questions, a breakdown of each criterion can found in Hong et al. (2018).”
Rêgo, A. D. S., Filipe, L., Dias, R. A., Alves, F. S., Queiroz, J., Ainla, A., Arruda, L. M., Fangueiro, R. M. E. S., Bouçanova, M., Bernardes, R. A., et al. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20054039 · Cited by 13
Telehealth in palliative care during the COVID ‐19 pandemic: A systematic mixed studies review
Xu, X., Ho, M., & Lin, C. (2023) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12637 · Cited by 11
Haupt, M. (2023) · Journal of Population Ageing · doi:10.1007/s12062-023-09420-x · Cited by 10
“The Mixed Meth-ods Appraisal Tool (MMAT) was used to assess the quality of the included studies (Hong et al., 2018).”
Geertz, W., Frerichs, W., Inhestern, L., & Bergelt, C. (2023) · Patient Education and Counseling · doi:10.1016/j.pec.2023.107844 · Cited by 16
Merry, K. J., MacPherson, M., Vis‐Dunbar, M., Whittaker, J. L., Silbernagel, K. G., & Scott, A. (2023) · Physical Therapy in Sport · doi:10.1016/j.ptsp.2023.06.002 · Cited by 15
Sattar, S. M. R. U., Akeredolu, O., Bogren, M. U., Erlandsson, K., & Borneskog, C. (2023) · Sexual & Reproductive Healthcare · doi:10.1016/j.srhc.2023.100917 · Cited by 10
“The MMAT is a critical appraisal tool that includes 25 criteria in five study categories that include qualitative, quantitative, and mixed-methods studies [17].”
MethodDavis, E. R., Aggar, C., & Bissett, M. (2023) · Journal of Clinical Nursing · doi:10.1111/jocn.16891 · Cited by 12
Occupational stress in primary care workers during the COVID-19 pandemic: mixed methods study
Tamborini, M. M. D. F., Centenaro, A. P. F. C., Souto, E. N. D. S., Andres, A. T. G., Stumm, E. M. F., & Colet, C. D. F. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6797.4041 · Cited by 18
Long-term health-related quality of life in meningioma survivors: A mixed-methods systematic review
Frances, S. M., Murray, L. J., Nicklin, E., Velikova, G., & Boele, F. W. (2024) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdae007 · Cited by 15
“See Supplementary Material 2 for MMAT scores.”
MethodThe homeless period: a qualitative evidence synthesis
Orsini, G. D., Tarabay, J., Hardy‐Johnson, P., Barker, S. L., & Greenway, F. T. (2024) · Women & Health · doi:10.1080/03630242.2024.2310716 · Cited by 15
“To assess the quality of the included studies, the Mixed-Method Appraisal Tool (MMAT; Hong et al. 2018) Sections 1 and 5, and the enhancing transparency in reporting the synthesis of qualitative research tools (ENTREQ; Tong et al. 2012) were chosen for quality assessment as the studies in this…”
MethodDahlke, S., Rayner, J., Fetherstonhaugh, D., Butler, J. I., & Kennedy, M. (2024) · International Journal of Nursing Education Scholarship · doi:10.1515/ijnes-2023-0042 · Cited by 19
Nystedt, T. A., Herder, T., Agardh, A., & Asamoah, B. O. (2024) · Global Health Action · doi:10.1080/16549716.2024.2340114 · Cited by 12
“Scoring is discour-aged using this tool; instead, the appraisal should be used to inform the quality of the included studies [18].”
MethodStargatt, J., Bhar, S. S., Bhowmik, J., & Mahmud, A. A. (2019) · JMIR Research Protocols · doi:10.2196/15512 · Cited by 12
“To assess the methodological quality of the included studies, the Mixed Methods Appraisal Tool (MMAT) will be used [30].”
MethodSchmidt, S., Nexø, M. A., Nørgaard, O., Willaing, I., Pedersen‐Bjergaard, U., Skinner, T. C., & Nørgaard, K. (2020) · Diabetic Medicine · doi:10.1111/dme.14347 · Cited by 11
Effect of yoga and meditation on tinnitus: a systematic review
Gunjawate, D., & Ravi, R. (2021) · Journal of Laryngology and Otology · doi:10.1017/s0022215121000566 · Cited by 10
Quality of life measures in pediatric multiple sclerosis: a systematic review and meta‐analysis
Ow, N., Mozafarinia, M., & Mayo, N. E. (2021) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.14870 · Cited by 15
Home fall of elderly people: implications of stressors and representations in the COVID-19 context
Santos, J. D. C., Arreguy-Sena, C., Pinto, P. F., et al. (2021) · Revista gaúcha de enfermagem · doi:10.1590/1983-1447.2021.20200221 · Cited by 10
“Mixed method study with concomitant triangulation(10) and data mixing integrating qualitative approaches (Social Representations Theory in structural(7) and procedural approaches(6) and BettyNeuman’s Nursing Theory(9)) and quantitative(11) (sectional study), presented according to criteria proposed by the Mixed Method…”
MethodDefining and Describing Terminal Ulcers in Adults at End of Life: An Integrative Review
Latimer, S. L., Walker, R., Ray‐Barruel, G., Shaw, J., Mackrell, K., Hunt, T., & Gillespie, B. M. (2021) · Advances in Skin & Wound Care · doi:10.1097/01.asw.0000798032.98853.95 · Cited by 10
“The design of each study was evaluated against five criteria using Y = yes, N = no, and CT = cannot tell.(38) Two researchers independently appraised each article and then discussed the overall strengths and weaknesses using the Mixed Methods Assessment Tool,(38) with discrepancies adjudicated by an experienced resear…”
MethodRelationships between Cognitive Functioning and Powered Mobility Device Use: A Scoping Review
Pellichero, A., Kenyon, L. K., Best, K. L., Lamontagne, M., Lavoie, M. D., Sorita, É., & Routhier, F. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182312467 · Cited by 11
“Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry, Canada) [18].”
BackgroundMilajerdi, H. R., Ordooiazar, F., & Dewey, D. (2022) · Child Neuropsychology · doi:10.1080/09297049.2022.2046721 · Cited by 21
“Mixed-methods studies were rated on the criteria for qualitative, quantitative, and mixed-methods studies (Hong et al., 2018).”
MethodThe wiki toolkit for planning, conducting and reporting mixed studies reviews
Pluye, P., Hong, Q. N., Granikov, V., & Vedel, I. (2018) · Education for Information · doi:10.3233/efi-180220 · Cited by 10
“More information about one critical appraisal tool that was developed specially for use in MSR is provided in another paper of this series (Hong et al., 2018): the Mixed Methods Appraisal Tool (MMAT) (version 2018).”
Self-citationSweeney‐Magee, M., Moustaqim‐Barrette, A., Gotay, C. C., & Dummer, T. (2020) · Journal of Advanced Nursing · doi:10.1111/jan.14389 · Cited by 10
Sasere, O. B., & Matashu, M. (2025) · Education Sciences · doi:10.3390/educsci15091168 · Cited by 9
“The MMAT was chosen for its versatility in appraising qualitative, quantitative, and mixed-methods studies using a comprehensive set of criteria (Hong et al., 2018).”
Adherence to exercise rehabilitation programmes in stroke survivors: a scoping review
Gwynne-Mayer, N., Drummond, A. E., Hancox, J., Alrubaia, W., & Taylor, I. M. (2025) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2024-002102 · Cited by 9
“The MMAT is a useful tool to evaluate the quality of reports with different methods.”
InfluentialDigital reflective practice among new graduate nurses – A scoping review
Bembridge, E., Crowfoot, G., Wilson, R. L., & Reis, J. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106688 · Cited by 13
Okafor, N. M., Thompson, I., Venkat, V., Robinson, C., Rao, A., Kulkarni, S., Frerichs, L., Ndiaye, K., Adenikinju, D., Iloegbu, C., et al. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000771 · Cited by 17
“We conducted a rigorous assessment of the risk of bias and the overall quality of the included articles, adhering to the guidelines outlined in the work by Hong et al. (2018) [17].”
Mental models and driver takeover in automated vehicles: a systematic review
Nasser, G., Morrison, B. W., & Wiggins, M. W. (2025) · Ergonomics · doi:10.1080/00140139.2025.2514599 · Cited by 13
“the methodological quality of the studies was assessed using the Mixed Methods appraisal tool (MMat; hong et al. 2018). the MMat is designed to evaluate the quality of qualitative, quantitative, and mixed-methods studies using five criteria for each study design. each criterion was rated as ‘yes,’…”
Cornelius, G., Hodgson, W., Maguire, R., & Egan, K. J. (2025) · Journal of Medical Internet Research · doi:10.2196/65385 · Cited by 18
“Methodological quality was evaluated with the relevant studies grouped by design type for the appropriate Critical Appraisal Skills Program (CASP) checklist [32] and the Mixed Methods Appraisal Tool (MMAT) assessment [33].”
Roychowdhury, S., Lanfranchi, V., & Mazumdar, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110944 · Cited by 21
Christensen, R., Fernandez, C. C., Topalian, N., Vaughan, L., Krenz, K., Pitman, A., & Fett, A.-K. J. (2025) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2025.105487 · Cited by 11
Systematic analysis of cutting-edge technology for the wellbeing and safety of older persons
Elsheikh, A., Othman, A., & Al‐Thani, D. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2559185 · Cited by 11
“the methodological rigor of the studies incorporated into this review was evaluated using the Mixed Methods appraisal tool (MMat) version 2018 [29]. this tool was selected due to its proven suitability for systematically assessing the quality of studies across a range of empirical designs,…”
Salam, R. A., Dunlop, K. L. A., Gide, T. N., Wilmott, J. S., Smith, A. L., & Cust, A. E. (2025) · JCO Precision Oncology · doi:10.1200/po-25-00063 · Cited by 14
Sofroniou, A., Patel, M. H., Premnath, B., & Wall, J. A. (2025) · Education Sciences · doi:10.3390/educsci15111544 · Cited by 51
“To assess the methodological quality of the 241 studies included in this meta-analysis, we utilised the Mixed-Methods Appraisal Tool (MMAT, Version 2018), which is specifically designed to evaluate qualitative, quantitative, mixed-methods and design-based research (Hong et al., 2018).”
Leon, R. J., Moroney, T., Fields, L., et al. (2022) · Contemporary Nurse · doi:10.1080/10376178.2022.2107040 · Cited by 9
“A total of 24 papers, of which 12 were assessed as high quality against the MMAT (Hong et al., 2018), were included in the review.”
BackgroundMethodEvidence Linking Stunting to Economic Outcomes in the SAARC Region: A Systematic Review
Seema, M., Akhlaq, A., Ali, H., & Nasser, D. (2022) · Pakistan Journal of Public Health · doi:10.32413/pjph.v12i2.978 · Cited by 12
“The authors utilized (28) Mixed Method Assessment Tool (MMAT) to measure quality.”
MethodThani, S. K. S. O., Hussein, H., & Ng, S. (2023) · Journal of Leisure Research · doi:10.1080/00222216.2023.2225058 · Cited by 10
“The authors developed the quality assessment checklist by adapting the quality appraisal tools such as the AXIS Tool (Downes et al., 2016), Mixed Method Appraisal Tool (Hong et al., 2018), and the Delphi Checklist (Verhagen et al., 1998).”
MethodBrummel, B., van Heumen, C., Smits, A., van den Berg, M. G. A., Ezendam, N. P. M., Pijnenborg, J. M. A., de van der Schueren, M. A. E., Wilkinson, S. A., & van der Meij, B. S. (2023) · Maturitas · doi:10.1016/j.maturitas.2023.107801 · Cited by 11
Staff training in physical interventions: a literature review
McDonnell, A., O’Shea, M. C., Bews-Pugh, S. J., McAulliffe, H., & Deveau, R. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1129039 · Cited by 13
“Included studies were further checked for methodological quality by using the Mixed Methods Appraisal Tool (MMAT) and an overall quality score between 1 (indicating relatively poor quality) and 5 (indicating relatively high quality) was assigned to each study (42).”
MethodWeiner‐Light, S., Tomasino, F., Wescott, A., Sideman, A. B., VELA, A. M., Possin, K. L., Penedo, F. J., & Wolf, M. S. (2023) · Aging & Mental Health · doi:10.1080/13607863.2023.2268050 · Cited by 14
“Because the MMAT was designed to appraise quantitative, qualitative, and mixed methods research, the authors discour-age the calculation of an overall score from the ratings of each criterion; rather, the authors suggest providing a more detailed presentation of the ratings of each criterion to better inform the quali…”
MethodInfluentialMahno, N. E., Tay, D. D., Khalid, N. S. A., Yassim, A. S. M., Alias, N., Termizi, S. A., Kasian, J., Mokhtar, N. M., & Ahmad, H. F. (2023) · Indian Journal of Microbiology · doi:10.1007/s12088-023-01135-z · Cited by 14
“The Mixed Methods Assessment Tool (MMAT) [26] was used to appraise the quality of included study.”
Xiong, B., Bailey, D. X., Prudon, P., Pascoe, E. M., Gray, L., Graham, F. A., Henderson, A. J., & Martin‐Khan, M. (2023) · International Journal of Nursing Sciences · doi:10.1016/j.ijnss.2023.11.001 · Cited by 13
“The MMAT includes a set of two screening questions and 25 criteria, with 5 criteria dedicated to each type of study design [39].”
BackgroundMethodNutritional counseling in childhood and adolescence: a systematic review
Neri, L. D. C. L., Guglielmetti, M., Fiorini, S., et al. (2024) · Frontiers in Nutrition · doi:10.3389/fnut.2024.1270048 · Cited by 9
Coombs, N. M., Porter, J. E., & Barbagallo, M. S. (2024) · Public Health · doi:10.1016/j.puhe.2023.12.023 · Cited by 13
Carbohydrate restriction during lactation: A systematic review
de Amorim, A. L. B., Rodrigues, E. F., Sussi, E. L., & Neri, L. D. C. L. (2024) · Nutrition Research · doi:10.1016/j.nutres.2024.02.007 · Cited by 12
Zhu, D., Zhao, J., Wang, M., Cao, B., Zhang, W., Li, Y., Zhang, C., & Han, T. (2024) · JMIR mhealth and uhealth · doi:10.2196/53798 · Cited by 15
“The Mixed Methods Assessment Tool (MMAT) was used to evaluate the methodological quality of the included studies [30].”
MethodDigennaro, S., & Tescione, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1390583 · Cited by 34
Andersson, C., Carlström, C., Amroussia, N., & Lindroth, M. (2024) · Sexual Health & Compulsivity · doi:10.1080/26929953.2024.2339208 · Cited by 11
“The eight empirical studies were read and assessed according to the Mixed Methods Appraisal Tool, (MMAT), a critical appraisal tool used for the appraisal stage in a systematic mixed studies review (Hong et al., 2018).”
MethodHolloway, H., Conroy, B., Isbel, S. T., & D'Cunha, N. M. (2024) · Digital Health · doi:10.1177/20552076241249568 · Cited by 14
“55 The article by Gruber et al. (2022) received the lowest score of one (20”
BackgroundMethodDue, C., Gartley, T., & Ziersch, A. M. (2024) · Australian Psychologist · doi:10.1080/00050067.2024.2343745 · Cited by 11
“Notably, the MMAT guide (Hong et al., 2018) explicitly discourages exclusion of studies based on quality.”
BackgroundMethodTobacco-free university campus policies: A systematic review
Geindreau, D., Girault, A., & Gallopel‐Morvan, K. (2024) · Journal of American College Health · doi:10.1080/07448481.2024.2367999 · Cited by 14
“29 One author used Excel© to score the 5 methodological quality criteria for each study (Supplementary Materials IV).”
Why We Share: A Systematic Review of Knowledge-Sharing Intentions on Social Media
Hu, J., & Noor, S. M. (2024) · Behavioral Sciences · doi:10.3390/bs14080636 · Cited by 31
Lived Experiences of Migrant Fathers in the Perinatal Period: A Systematic Review and Analysis
Vo, H. N., McKenzie‐McHarg, K., Bennett, P. C., & Mai, D. L. (2024) · Journal of Immigrant and Minority Health · doi:10.1007/s10903-024-01627-0 · Cited by 12
“Author Vo evaluated the quality of the 14 selected stud - ies using the Mixed Methods Appraisal Tool (MMAT; Hong et al., [26 ]).”
Association between the Indoor Visual Environment and Cognition in Older Adults: A Systematic Review
Luo, K., Wei, D., He, B., & Fei, Y. (2024) · Buildings · doi:10.3390/buildings14103066 · Cited by 11
Sorrentino, M., Fiorilla, C., Mercogliano, M., Esposito, F., Stilo, I., Affinito, G., Moccia, M., Lavorgna, L., Salvatore, E., Maida, E., et al. (2024) · Frontiers in Neurology · doi:10.3389/fneur.2024.1474336 · Cited by 15
Impact of climate change on child outcomes: an evidence gap map review
Meherali, S. M., Aynalem, Y. A., Nisa, S., Kennedy, M., Salami, B. O., Adjorlolo, S., Ali, P. A., Silva, K. L., Aziato, L., Richter, S. M., et al. (2024) · BMJ Paediatrics Open · doi:10.1136/bmjpo-2024-002592 · Cited by 10
“Quality ratings were assigned using the Mixed Methods Appraisal Tool 25 and Assessment of Multiple Systematic Reviews 2.”
Liguori, J., Osei-Kwasi, H., Savy, M., Nanema, S., Laar, A., & Holdsworth, M. (2024) · Public Health Nutrition · doi:10.1017/s1368980024001939 · Cited by 9
“This tool was designed to critically appraise multiple types of research methodologies in systematic reviews (19,20) .”
Career Adaptability Across Southeast Asia: A Systematic Review
Kumar, D. A., Lau, P. L., Chua, K. H., Voon, S. P., & Lo, Y. Y. (2024) · SAGE Open · doi:10.1177/21582440241289737 · Cited by 24
“The Mixed Method Appraisal Tool (MMAT) (Hong et al., 2018) was employed to conduct the quality assessment.”
InfluentialKanmodi, K. K., Salami, A. A., Shah, K., Zohoori, F. V., & Nnyanzi, L. A. (2024) · Health Science Reports · doi:10.1002/hsr2.70171 · Cited by 9
“All the included original research articles were assessed for risk of bias (quality) using the Mixed Methods Appraisal Tool (MMAT) version 2018 [42].”
BackgroundMethodInfluentialCastro, Ó., Salamanca-Sanabria, A., Alattas, A., Teepe, G. W., Leidenberger, K., Fleisch, E., Car, L. T., Müller‐Riemenschneider, F., & Kowatsch, T. (2024) · Archives of Public Health · doi:10.1186/s13690-024-01424-z · Cited by 11
“The MMAT is a valid and reliable tool that allows appraisal of five different categories of studies: qualitative research, RCTs, non-randomised studies, quantitative descriptive studies, and mixed methods studies [47].”
MethodPu, L., Gamage, M. W. K., Li, N., et al. (2024) · Geriatric Nursing · doi:10.1016/j.gerinurse.2024.10.047 · Cited by 9
Brain–computer interfaces patient preferences: a systematic review
Brannigan, J., Liyanage, K., Horsfall, H. L., Bashford, L., Muirhead, W. R., & Fry, A. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad94a6 · Cited by 18
“Study quality was appraised using the Mixed Methods Appraisal Tool [8], and no further studies were excluded following appraisal (supplementary data 3).”
Moghbeli, G., Roshangar, F., Soheili, A., Ahmadi, F., Feizollahzadeh, H., & Hassankhani, H. (2024) · International Journal of Emergency Medicine · doi:10.1186/s12245-024-00788-x · Cited by 10
Zarekar, M., Al-Shehabi, H., Dörner, R., Weishaar, H. B., Lennemann, T., Bcheraoui, C. E., & Bernasconi, A. R. (2024) · EBioMedicine · doi:10.1016/j.ebiom.2024.105520 · Cited by 17
Blütgen, S., Pralong, A., Wilharm, C., Eisenmann, Y., Voltz, R., & Simon, S. T. (2025) · BMC Palliative Care · doi:10.1186/s12904-025-01670-0 · Cited by 12
Kay, J., Elliott, S., Crossman, S., Drummond, M., & Petersen, J. M. (2025) · Sport in Society · doi:10.1080/17430437.2025.2460486 · Cited by 12
“Two independent reviewers (JK and SC) assessed the scientific and methodological quality and rigour of the included studies using the Mixed Methods Appraisal Tool (MMAT; Hong et al. 2018).”
Usarralde, M. J. M., Conchell, R., Herrero, M. V., & Pérez-Tabernero, L. (2025) · Behavioral Sciences · doi:10.3390/bs15030245 · Cited by 21
“…authors by e-mail when we needed Disagreements were resolved by consensus with a third author. additional information for our review or to determine eligibility We used the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018; Pace et al., 2012) to assess the methodological quality of the studies.”
Webb, C., Dabkowski, E., Missen, K., & Missen, A. (2025) · Journal of Clinical Nursing · doi:10.1111/jocn.17721 · Cited by 18
“The Mixed Methods Appraisal Tool (MMAT) was used to evaluate studies of a mixed-methods design (Hong et al. 2018), with the Joanna Briggs Institute (JBI) checklist for quasi-experimental studies (Tufanaru et al. 2020) and the JBI checklist for analytical cross-sectional studies (Moola et al. 2020)…”
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.