The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 901–1,000 of 4,578
Vargas, J., Smith, M., Chipchase, L. S., & Morris, M. E. (2024) · Advances in Health Sciences Education · doi:10.1007/s10459-024-10342-2 · Cited by 29
“The MMAT assesses methodological quality for categories: (i) qualitative (ii) quantitative randomised controlled trial (RCT) (iii) quantita - tive non-RCT (iv) quantitative descriptive (v) mixed methods.”
Adombire, S., Baiden, D., Puts, M. T. E., Ritchie, L. M. P., Ani–Amponsah, M., & Cranley, L. A. (2024) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12734 · Cited by 23
Stroeken, Y., Hendriks, F. J., Beltman, J. J., & ter Kuile, M. (2024) · Cancers · doi:10.3390/cancers16203456 · Cited by 22
“Fourteen articles were graded according to MMAT criteria [33] (see The time between diagnosis and study period varied from diagnosis [21] to eleven years post-diagnosis [24].”
MethodNowrouzi‐Kia, B., Chan, H., Zhu, S., Nandan, S., Bani‐Fatemi, A., Howe, A. S., Gross, D. P., Gohar, B., Yazdani, A., & Chattu, V. K. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241289436 · Cited by 19
“Additionally, The Mixed Methods Appraisal Tool was used to appraise mixed methods studies (Hong et al., 2018).”
Kip, A., Valencia, S., Glunz, E., Lowe, S. R., Tam, K., & Nexhmedin Morina, B. (2024) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796024000672 · Cited by 21
“…of bias appraisal tool based on the Risk of Bias Assessment Tool for Non-randomized Studies (Kim et al. , 2013), the Mixed Methods Appraisal Tool (Hong et al. , 2018) and the Strengthening the Reporting of Observational Studies in Epidemiology guidelines (von Elm et al. , 2007) including…”
Critical literacy in artificial intelligence assisted writing instruction: A systematic review
Arseven, T., & Bal, M. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101850 · Cited by 42
Malkowski, O. S., Harvey, J., Townsend, N. P., Kelson, M. J., Foster, C. E., & Western, M. J. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01753-4 · Cited by 34
“To account for the range of study designs identified and included in the systematic review (when considered in tandem with the quantitative evidence, which is reported separately), the Mixed Methods Appraisal Tool (MMAT) was used to assess risk of bias [25].”
A Systematic Literature Review of Pedestrian Safety in Urban Transport Systems
Sharma, S. N., & Dehalwar, K. (2025) · Journal of Road Safety · doi:10.33492/jrs-d-25-4-2707507 · Cited by 40
Li, C. G., Cui, H., & Hagedorn, L. S. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100571 · Cited by 104
Dillon, C. B., Saab, M. M., Meehan, E., Goodwin, J., Murphy, M., Heffernan, M., Greaney, S., Kilty, C., Hartigan, I., Chambers, D., et al. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.07.113 · Cited by 28
Kristjánsdóttir, Ó. B., Vågan, A., Svavarsdóttir, M. H., Børve, H. B., Hvinden, K., Duprez, V., Hecke, A. V., Heyn, L. G., Strømme, H., & Stenberg, U. (2020) · Patient Education and Counseling · doi:10.1016/j.pec.2020.12.006 · Cited by 26
Illness perceptions of adults with eczematous skin diseases: a systematic mixed studies review
Rocholl, M., Ludewig, M., Brakemeier, C., John, S. M., & Wilke, A. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01687-5 · Cited by 23
“Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. Mixed Methods Appraisal Tool (MMAT), version 2018.”
MethodInfluentialManagement of Pediatric Atopic Dermatitis by Primary Care Providers: A Systematic Review
Young, T. K., Glick, A. F., Yin, H. S., Kolla, A. M., Velazquez, J. J., Nicholson, J., & Oza, V. S. (2021) · Academic Pediatrics · doi:10.1016/j.acap.2021.07.008 · Cited by 25
Jedliński, M., Janiszewska‐Olszowska, J., Mazur, M., Ottolenghi, L., Grocholewicz, K., & Galluccio, G. (2021) · Coatings · doi:10.3390/coatings11121488 · Cited by 19
“The response ‘No’ to two of the screening questions or ‘Can’t tell’ to one or both the screening questions might indicate that the paper cannot be appraised using the MMAT.”
MethodInfluentialDigital Tools in Behavior Change Support Education in Health and Other Students: A Systematic Review
Gosak, L., Štiglic, G., Cilar, L., Félix, I. B., Braam, K. I., Fijačko, N., Guerreiro, M. P., & Lorber, M. (2021) · Healthcare · doi:10.3390/healthcare10010001 · Cited by 19
“We reported our MMAT results in metrics and not as no metrics, as is described in the MMAT instructions, because of that way being more informative for the readers (Table S5) [29,30].”
MethodResilience-Promoting Programs in Families of Children With Cancer: A Systematic Review
Park, M., Choi, E. K., Lee, H., Park, H. E., & Chinbayar, A. (2022) · Journal of Pediatric Hematology/Oncology Nursing · doi:10.1177/27527530211055997 · Cited by 25
“The MMAT measures the quality of literature as “Yes,” “No,” or “Can’t tell” according to methodological quality criteria (Hong et al., 2018).”
MethodTibbs, M., O’Reilly, A., Dwan-O'Reilly, M., & Fitzgerald, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061084 · Cited by 24
Stehr, P., Reifegerste, D., Rossmann, C., Caspar, K., Schulze, A., & Lindemann, A. (2022) · Patient Education and Counseling · doi:10.1016/j.pec.2022.04.015 · Cited by 23
Orth, Z., & van Wyk, B. E. (2022) · Systematic Reviews · doi:10.1186/s13643-022-01961-0 · Cited by 31
“Firstly, the Mixed Methods Appraisal Tool (MMAT) (Additional file 1: Appendix A) will be used to assess the methodological quality of the studies as it allows for summarising the overall quality across a range of study designs [31].”
MethodToh, G., Pearce, E., Vines, J., Ikhtabi, S., Birken, M., Pitman, A., & Johnson, S. (2022) · BMC Psychiatry · doi:10.1186/s12888-022-03889-0 · Cited by 22
“Following previous scoping reviews that also included quality assessment [45–50], we used the Mixed Methods Appraisal Tool (MMAT) [51] for quality assessment due to the inclusion of quantitative and mixed methods study in this review.”
The use of formal care for dementia from a professional perspective: a scoping review
Bergmann, S., Peper, J., & Bieber, A. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08229-2 · Cited by 25
“The results of the critical appraisal of the included studies with the MMAT measurement are presented in Table 2.”
MethodInfluentialExploring the Efficacy of Social Work Interventions in Hospital Settings: A Scoping Review
Petruzzi, L. J., Ewald, B., Covington, E., Rosenberg, W., Golden, R., & Jones, B. L. (2022) · Social Work in Public Health · doi:10.1080/19371918.2022.2104415 · Cited by 19
“The quality of included studies was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodHalim, A. F. N. A., Ahmad, D., Ling, M. Y. J., Masdor, N. A., Ramly, N., Othman, R., Kandayah, T., Hassan, M. R., & Dapari, R. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191912971 · Cited by 26
“It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (1) quantitative, (2) qualitative, (3) randomized controlled, (4) nonrandomized, and (5) mixed methods [18].”
BackgroundMethodInfluentialAbobotulinumtoxinA Doses in Upper and Lower Limb Spasticity: A Systematic Literature Review
Schnitzler, A., Dince, C. C., Freitag, A. P., et al. (2022) · Toxins · doi:10.3390/toxins14110734 · Cited by 19
Predictors of smoking cessation outcomes identified by machine learning: A systematic review
Bickel, W. K., Tomlinson, D. C., Craft, W. H., Ma, M., Dwyer, C. L., Yeh, Y., Tegge, A. N., Lemos, R. F., & Athamneh, L. N. (2023) · Addiction Neuroscience · doi:10.1016/j.addicn.2023.100068 · Cited by 33
Duval, D., Pearce‐Smith, N., Palmer, J. C., Sarfo-Annin, J. K., Rudd, P., & Clark, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02219-z · Cited by 19
“Abbreviations AHRQ Agency for Healthcare Research and Quality CASP Critical Appraisal Skills Programme JBI Joanna Briggs Institute MMAT Mixed Methods Appraisal Tool NHLBI National Heart, Lung and Blood Institute NICE National Institute for Health and Care Excellence N/A Not applicable PHE Public Health England QCC Qua…”
InfluentialEmergency care in post-conflict settings: a systematic literature review
Werner, K., Kak, M., Herbst, C. H., & Lin, T. K. (2023) · BMC Emergency Medicine · doi:10.1186/s12873-023-00775-0 · Cited by 27
Benefits of Hearing Voices Groups & Other Self-Help Groups for Voice Hearers: A Systematic Review
Corentin, C., Fitzgerald, C., & Goodwin, J. (2023) · Issues in Mental Health Nursing · doi:10.1080/01612840.2023.2189953 · Cited by 32
“The mixed-method study was appraised using then MMAT version 18 (Hong et al., 2018); of the seven questions asked four were answered “yes”, one “Can’t tell”, and two “no”.”
MethodLing, M. Y. J., Ahmad, N., & Aizuddin, A. N. (2023) · PLoS ONE · doi:10.1371/journal.pone.0286518 · Cited by 40
“However, due to problems faced by researchers in reporting the MMAT results, suggestion was provided for reporting of an overall score (5*****/100% quality criteria met; 4****/80% quality criteria met; 3***/60% quality criteria met; 2**/40% quality criteria met; 1*/20% quality criteria met).”
MethodInfluentialKelly, Y., O’Rourke, N., Flynn, R., O’Connor, L., & Hegarty, J. M. (2023) · BMJ Quality & Safety · doi:10.1136/bmjqs-2022-015287 · Cited by 30
“Methodological quality was assessed using the Critical Appraisal Skills Programme tool 25 for qualitative studies, Joanna Briggs Institute Critical Appraisal Tools 26 for quantitative studies and The Mixed Methods Appraisal Tool 27 for mixed methods studies.”
Rizvi, F., Wilding, H., Rankin, N. M., Gautier, R. L., Gurren, L., Sundararajan, V., Bellingham, K., Chua, J., Crawford, G. B., Nowak, A. K., et al. (2023) · Palliative Medicine · doi:10.1177/02692163231186177 · Cited by 27
Kafle, E., Brooks, C. P., Chawner, D., Foye, U., Declercq, D., & Brooks, H. L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1161703 · Cited by 45
“Of these studies, one was deemed high quality and met all MMAT criteria (Konradt et al., 2013).”
MethodInfluentialBlanchette, V., Patry, J., Brousseau‐Foley, M., Todkar, S., Libier, S., Leclerc, A.-M., Armstrong, D. G., & Tremblay, M. (2023) · Frontiers in Endocrinology · doi:10.3389/fendo.2023.1177020 · Cited by 31
“However, two reviewers (VB and ST) conducted an appraisal based mixed methods appraisal tool (MMAT) and chose at posteriori according to the studies included (43).”
Barriers and facilitators of conducting research with team science approach: a systematic review
Ghamgosar, A., Nemati‐Anaraki, L., & Panahi, S. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04619-0 · Cited by 30
“The ‘can’t tell’ response category indicates that the article does not report appropriate information to answer ‘yes’ or ‘no’ or that it reports unclear information related to the criterion [27].”
BackgroundA systematic review of smartphone applications for cancer survivors
Changrani, K., Chima, S., Sharma, A. K., Han, G.-G., Sharma, A., McNamara, M., Jefford, M., Emery, J. D., & Druce, P. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-023-01435-9 · Cited by 23
“Majority of the studies did not specify which operating platform the apps were available on; 4 were available for both iPhone (iOS) and Android users [28, 45, 48, 51], 3 were available exclusively on Android devices [33, 44, 52], and 4 were available exclusively on iPhone devices [34, 40, 42].”
What do we currently know about the development of talent? A systematic review in the soccer context
Verbeek, J., van der Steen, S., Yperen, N. W. V., & Hartigh, R. J. R. D. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2023.2283874 · Cited by 29
“In addition, all studies included were assessed using the updated version of the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Risk assessment for people living with dementia: a systematic review
Hoe, J., Profyri, E., Kemp, C., Manela, M., Webster, L., Anthony, J., Costafreda, S. G., Arrojo, F., Souris, H., & Livingston, G. (2023) · International Psychogeriatrics · doi:10.1017/s1041610223004398 · Cited by 27
“Two reviewers (EP and CK) independently assessed the quality of the included studies using the Mixed Method Appraisal Tool (MMAT: Hong et al ., 2018), and any discrepancies were resolved by consulting a third reviewer (JH).”
MethodRowe, O., & Nadkarni, A. (2023) · Cambridge Prisms Global Mental Health · doi:10.1017/gmh.2023.91 · Cited by 37
“Full texts were critically appraised using the McGill University Mixed Methods Appraisal Tool (Appendix 4) and no texts were excluded on this basis (Hong et al., 2018).”
MethodHuman Health Adaptation Strategies to Climate-Induced Extreme Weather Events: A Systematic Review
Amnuaylojaroen, T., & Parasin, N. (2024) · Earth · doi:10.3390/earth5040038 · Cited by 19
“To ensure the inclusion of high-quality studies, we applied a rigorous methodological quality filter using the Mixed Methods Appraisal Tool (MMAT) [21].”
Xie, J., Hunter, A., Biesty, L., & Grealish, A. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104814 · Cited by 21
Restrição do acompanhamento de lactentes prematuros na pandemia da COVID-19: abordagem mista
Reichert, A. P. D. S., Soares, A. R., Guedes, A. T. A., Brito, P. K. H., Bezerra, I. C. D. S., Santos, N. C. C. D. B., & Collet, N. (2022) · Acta Paulista de Enfermagem · doi:10.37689/acta-ape/2022ao022066 · Cited by 21
The effectiveness of interventions to reduce cancer‐related stigma: An integrative review
Zheng, S., Liu, S., Yang, Q., Chan, S. W., Huang, W., Jiang, X., & Zhu, J. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17014 · Cited by 20
Kitto, S. C., Fantaye, A. W., Zevin, B., Fowler, A. C., Sachdeva, A. K., & Raîche, I. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.02.011 · Cited by 25
Alodan, H. A., Sutt, A., Hill, R., Alsadhan, J., & Cross, J. (2024) · Australian Critical Care · doi:10.1016/j.aucc.2024.04.006 · Cited by 22
Factors Influencing Nurses Turnover in Saudi Arabia: A Systematic Review
Albalawi, A., Pascua, G. P., Alsaleh, S. A., Sabry, W., Ahajan, S. N., Abdulla, J., Abdulalim, A., & Salih, S. S. (2024) · Nursing Forum · doi:10.1155/2024/4987339 · Cited by 31
Ein, N., Gervasio, J., Cyr, K. S., Liu, J. J. W., Baker, C., Nazarov, A., & Richardson, J. D. (2024) · Frontiers in Health Services · doi:10.3389/frhs.2024.1426202 · Cited by 38
“Hong et al. (58) noted that calculating an overall score is not advisable.”
BackgroundMethodCatasta, A., Martini, C., Mersanne, A., et al. (2024) · Diagnostics · doi:10.3390/diagnostics14151658 · Cited by 18
“The quality of the studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [6].”
MethodA Systematic Review Uncovering Modifiable Influences on Statin Adherence
Rosenberg, J., Lampridou, S., Moores, A., Garfield, S., Wingfield, D., & Judah, G. (2025) · Patient Preference and Adherence · doi:10.2147/ppa.s502645 · Cited by 21
“25 The first reviewer appraised all studies, and a second reviewer (SL) randomly assessed 25% with a proportion of agreement of 0.71.”
InfluentialChia, A. W.-Y., Teo, W. L.-L., Acharyya, S., Munro, Y. L., & Dalan, R. (2025) · BMC Medicine · doi:10.1186/s12916-025-03873-x · Cited by 26
“The only mixed methods study included in this review was assessed by a reviewer using the mixed methods appraisal tool [75].”
Benjamin, L., Gillard, S., Nielsen, J. J., da Silva, M. E. P., & Sin, J. (2025) · Trauma Violence & Abuse · doi:10.1177/15248380251320982 · Cited by 31
“Quality assessment of included articles was independently rated by two reviewers (L.B. and M.C.E.S.) using The Mixed Methods Appraisal Tool (MMAT) v2018 (Hong et al., 2018).”
Defining open goals for the promotion of health behaviours: a critical conceptual review
Swann, C., Goddard, S. G., Schweickle, M. J., Hawkins, R. M., Williamson, O., Gargioli, D., Clarke, M. M., Jackman, P. C., & Vella, S. A. (2025) · Health Psychology Review · doi:10.1080/17437199.2025.2467695 · Cited by 23
“We used the mixed methods appraisal tool (MMAT; Hong et al., 2018) to assess the quality of the included studies, as the MMAT enables appraisal of quantitative, qualitative, and mixed-methods studies.”
Cipriani, G. E., Molfese, S., Giovannelli, F., Güntekin, B., Vitali, N., Marcato, R., & Amanzio, M. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106121 · Cited by 22
“Instead, the MMAT ratings for each criterion were detailed to better inform readers about the quality of the included studies (see Supplementary Table S1 for MMAT evaluations).”
InfluentialCai, C., Mei, Z., Yang, Y., & Luo, S. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1578198 · Cited by 32
Maio, M. D., Montenegro, S., Papale, O., et al. (2025) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2025.1566208 · Cited by 18
Chatzigeorgiou, C., Christou, E., & Simeli, I. (2025) · Administrative Sciences · doi:10.3390/admsci15090371 · Cited by 43
Zhan, X., & Zhu, W. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1663670 · Cited by 29
Using Large Language Models for Chronic Disease Management Tasks: Scoping Review
Mukalazi, S. H., Ouyang, J., Namatovu, H. K., Ssemaluulu, P. M., Kimbugwe, N., Orach, C. G., & Waiswa, P. (2025) · JMIR Medical Informatics · doi:10.2196/66905 · Cited by 26
“The Mixed Methods Appraisal Tool (MMAT, version 2018) [26] was used to perform a formal methodological quality assessment.”
Borthwick, C., O’Connor, R. C., & Kennedy, L. (2020) · Psychology and Health · doi:10.1080/08870446.2020.1772971 · Cited by 19
Chen, Y.-H., Jones, C., & Bannatyne, A. J. (2020) · Educational Gerontology · doi:10.1080/03601277.2020.1767883 · Cited by 22
Doctoral employability: A systematic literature review and research agenda
Young, S., Kelder, J., & Crawford, J. (2020) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2020.3.s1.5 · Cited by 18
“For the quality review, an adapted form of the Mixed Method Appraisal Tool (MMAT: Hong et al., 2018) was used with scores ranging from high (75% to 100%), medium (50% to 75%), and low (below 50%).”
MethodBalante, J., van den Broek, D., & White, K. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14814 · Cited by 21
Hawsawi, T., Stein‐Parbury, J., Orr, F., Roche, M. A., & Gill, K. H. (2021) · International Journal of Mental Health Nursing · doi:10.1111/inm.12908 · Cited by 21
The value of mixed methods work-family research for human resource management: a review and agenda
Jaga, A., & Guetterman, T. C. (2021) · The International Journal of Human Resource Management · doi:10.1080/09585192.2021.1964092 · Cited by 25
“Although we considered using a quality rating tool such as the Mixed Methods Appraisal Tool (Hong et al., 2018), the output would have limited our rating to “yes”, “no”, or “can’t tell” for each criterion and would not have provided the descriptive information we sought to understand how MMR is…”
MethodRandell, R., Wright, J. M., Alvarado, N., Healey, F., Dowding, D., Smith, H., Hardiker, N. R., Gardner, P. H., Ward, S., Todd, C. J., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-049765 · Cited by 22
“26 However, we will not exclude studies based on this appraisal.”
English, M., Wallace, L., Evans, J. R., Diamond, S., & Caperchione, C. M. (2021) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2021.101676 · Cited by 27
Gynell, I., Kemps, E., & Prichard, I. J. (2022) · Appetite · doi:10.1016/j.appet.2022.105997 · Cited by 25
Burrola-Mendez, Y., Kamalakannan, S., Rushton, P. W., Bouziane, S.-A., Giesbrecht, E., Kirby, R. L., Gowran, R. J., Rusaw, D. F., Tasiemski, T., Goldberg, M. R., et al. (2022) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2022.2037757 · Cited by 25
Health care providers’ responses to sexually abused children and adolescents: a systematic review
Rahnavardi, M., Shahali, S., Montazeri, A., & Ahmadi, F. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-07814-9 · Cited by 27
“Quality assessment and risk of Bias Studies were evaluated for scientific quality using the Mixed Methods Appraisal Tool (MMAT) version 2018 and its associated scoring system [19].”
MethodA systematic review of lifestyle and health among patrolling police officers
Valmari, E. G., Ghazinour, M., Nygren, U., & Gilenstam, K. (2022) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2022.2083013 · Cited by 34
“The majority of the studies were completed in Germany and Sweden, see Table 2 for detailed information on the research context, and population characteristics, as well as results from the quality assessment using MMAT.”
MethodInfluentialWhy we prescribe antibiotics for too long in the hospital setting: a systematic scoping review
Janssen, R. M. E., Oerlemans, A. J. M., van der Hoeven, J. G., ten Oever, J., Schouten, J. A., & Hulscher, M. (2022) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkac162 · Cited by 22
Gitonga, I., Desmond, D. M., Duda, N., & Maguire, R. (2022) · Psycho-Oncology · doi:10.1002/pon.6019 · Cited by 28
“The latest MMAT guidelines discourages presenting a single number to denote quality as it does not tell what specific study areas are problematic.(34) For this reason, interpretation took the following form: 4–5 criteria met = high quality, 2–3 criteria met = moderate quality, 0–1 criterion met = low quality, as per p…”
BackgroundMethodInfluentialAn Exploration of the Experiences of Substitute Teachers: A Systematic Review
Reupert, A., Sullivan, A. M., Tippett, N., White, S., Woodcock, S., Chen, L., & Simons, M. (2023) · Review of Educational Research · doi:10.3102/00346543221149418 · Cited by 19
“The Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) was employed as it allows for concomitant appraisal of qualitative, quantitative, and mixed-methods studies.”
MethodGregory, S., Caffrey, L., & Daly, D. (2023) · Women and Birth · doi:10.1016/j.wombi.2023.02.003 · Cited by 27
Madawala, S., Quach, A., Lim, J. Y., Varatharaj, S., Perera, B., Osadnik, C. R., & Barton, C. (2023) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2022-001514 · Cited by 19
“25 Studies were appraised for quality using the Mixed Methods Appraisal Tool (MMAT) by two independent reviewers (SM and JYL) (see online supplemental appendix 2 for results) 26 and disagreements were resolved through discussion.”
MethodDabkowski, E., Porter, J. E., Barbagallo, M. S., Prokopiv, V., Snell, C., & Missen, K. (2023) · Cogent Psychology · doi:10.1080/23311908.2023.2186327 · Cited by 23
“Mixed methods studies were appraised using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodImpact of Locus of Control on Patient–Provider Communication: A Systematic Review
Rodriguez, A., Patton, C. D., & Stephenson-Hunter, C. (2023) · Journal of Health Communication · doi:10.1080/10810730.2023.2192014 · Cited by 18
Knowledge and Practice of Personal Protective Measures Against COVID-19 in Africa: Systematic Review
Kawuki, J., Chan, P. S.-F., Fang, Y., Chen, S., Mo, P. K. H., & Wang, Z. J. (2023) · JMIR Public Health and Surveillance · doi:10.2196/44051 · Cited by 19
“We assessed the information from the included articles using the Mixed Methods Appraisal Tool (version 2018) with detailed descriptions of the rating [17].”
Integrating social support into interventions among the elderly in nursing homes: a scoping review
Behrendt, D., Spieker, S., Sumngern, C., & Wendschuh, V. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-071962 · Cited by 34
Osabi, L. A., van de Klundert, J. J., Alhurishi, S. A., & Cramm, J. M. (2023) · BMC Public Health · doi:10.1186/s12889-023-15725-5 · Cited by 23
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [16], with scores reflecting the number (0–5) of assessment criteria met.”
MethodInfluentialAlvarado, N., McVey, L., Wright, J. M., Healey, F., Dowding, D., Cheong, V., Gardner, P. H., Hardiker, N. R., Lynch, A., Zaman, H., et al. (2023) · BMC Geriatrics · doi:10.1186/s12877-023-04045-3 · Cited by 26
“Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The mixed methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers.”
MethodValidated Questionnaires in Flow Theory: A Systematic Review
Rosas, D. A., Padilla-Zea, N., & Burgos, D. (2023) · Electronics · doi:10.3390/electronics12132769 · Cited by 25
“In Annex B1, we summarize the MMAT [48], including each study supported by experimental work (thus, we omitted the first two screening questions, as reiterative) into the category of quantitative descriptive studies.”
BackgroundMethodInfluentialHealthy Behavior and Sports Drinks: A Systematic Review
Muñoz-Urtubia, N., Vega-Muñoz, A., Estrada-Muñoz, C., Salazar-Sepúlveda, G., Contreras-Barraza, N., & Castillo, D. (2023) · Nutrients · doi:10.3390/nu15132915 · Cited by 37
“Quantitative, qualitative, and mixed-study types were included (under MMAT quality criteria) [49].”
MethodIyamu, I. O., Sierra-Rosales, R., Estcourt, C. S., Salmon, A., Koehoorn, M. W., & Gilbert, M. (2023) · Sexually Transmitted Infections · doi:10.1136/sextrans-2023-055749 · Cited by 22
“However, we assessed the quality of the available evidence using the Mixed Methods Appraisal Tool 20 (online supplemental appendix B).”
Conflict of interest and funding in health communication on social media: a systematic review
Helou, V., Mouzahem, F., Makarem, A., Noureldine, H. A., El-Khoury, R., Oweini, D. A., Halak, R., Hneiny, L., Khabsa, J., & Akl, E. A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072258 · Cited by 20
Khanthavudh, C., Grealish, A., Tzouvara, V., Huang, J., & Leamy, M. C. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104591 · Cited by 21
Impact of mental health stigma on help-seeking in the Caribbean: Systematic review
Gallimore, J.-B., Díaz, K. G., Gunasinghe, C., Thornicroft, G., Salisbury, T. T., & Gronholm, P. C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0291307 · Cited by 35
Jafree, S. R., & Barlow, J. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076883 · Cited by 19
Maternity clinician use of shared decision‐making in antenatal care: A scoping review
Hawke, M., Considine, J., & Sweet, L. P. (2023) · Birth · doi:10.1111/birt.12805 · Cited by 23
Ambagtsheer, R. C., Borg, K., Townsin, L., de Plaza, M. A. P., O'Brien, L. M., Kunwar, R. S., & Lawless, M. (2024) · Archives of Gerontology and Geriatrics Plus · doi:10.1016/j.aggp.2024.100008 · Cited by 22
Nunn, J. L., Takashima, M., Wray-Jones, E. M., Raj, T. S., Hanna, D. M. T., & Ullman, A. J. (2024) · Supportive Care in Cancer · doi:10.1007/s00520-024-08853-0 · Cited by 32
“The Mixed Methods Appraisal Tool (MMAT) [23] was used by two reviewers to assess the quality of evidence for the studies included.”
MethodLai, D. J., Liu, Z., Johnston, E., Dikomitis, L., D’Oliveira, T. C., & Shergill, S. S. (2024) · Critical Care · doi:10.1186/s13054-024-05089-6 · Cited by 20
“Two reviewers (DL, and ZL) independently assessed the risk of bias and the quality of studies using the Mixed Methods Appraisal Tool (MMAT) [24].”
MethodAdvance Decision Making in Bipolar: A Systematic Review
Stephenson, L., Gergel, T., Gieselmann, A., Scholten, M., Keene, A. R., Rifkin, L., & Owen, G. S. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.538107 · Cited by 20
“The Mixed Methods Appraisal Tool (MMAT) was used to categorise the body of literature identified, further clarifying the current state of the available evidence base (10).”
MethodDecision-Making Experience Related to Mastectomy Among Women With Breast Cancer
Liu, J., Hunter, S., Zhu, J., Lee, R., & Chan, S. W. (2021) · Cancer Nursing · doi:10.1097/ncc.0000000000000981 · Cited by 17
“Data Evaluations This review adopted the Mixed Methods Appraisal Tool (MMAT) version 2018 for quality appraisal.(16) The MMAT was developed to evaluate the quality of qualitative, quantitative, and mixed-methods studies, including justification of research questions, data collection methods, data analysis, and outcome…”
MethodNyberg, A., Rajaleid, K., & Demmelmaier, I. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19116783 · Cited by 25
“Using the MMAT meant that an assessment of studies with different designs was parsimonious but possibly not thorough enough to capture all important aspects of methodological quality.”
MethodInfluentialPsychosocial interventions for ovarian cancer survivors: A systematic review
Yeoh, S. A., Webb, S., Phillips, A., Li, L. S. K., & Kumar, S. (2024) · Psycho-Oncology · doi:10.1002/pon.6280 · Cited by 17
Fotland, S.-L. S., Midtbø, V., Vik, J., Zakariassen, E., & Johansen, I. H. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02580-7 · Cited by 27
“Methodological limitations for each study were independently assessed and then discussed by SLSF and IHJ using the Mixed Methods Appraisal Tool (MMAT) [23].”
McDonald, K., Schwartz, A., Dinerstein, R. D., Olick, R. S., & Sabatello, M. (2024) · Disability and health journal · doi:10.1016/j.dhjo.2024.101669 · Cited by 19
Steller, R. S., Recklies, E. A., & Schweizer-Ries, P. (2024) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2024.06.422 · Cited by 23
Feeley, C., & Stacey, T. (2024) · Midwifery · doi:10.1016/j.midw.2024.104152 · Cited by 17
Tam, L. Y. C., Taechameekietichai, Y., & Allen, J. L. (2024) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-024-02590-9 · Cited by 35
“Considering the anticipated inclusion of diverse study designs, the intended quality assessment tool cited in the study protocol was the Mixed Methods Appraisal Tool (MMAT-2018) [59].”
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.