The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 2,301–2,400 of 4,578
Dzinamarira, T., Kuupiel, D., Vezi, P. N., & Mashamba-Thompson, T. P. (2021) · Public Health in Practice · doi:10.1016/j.puhip.2021.100177 · Cited by 8
Domingue, J., Jäger, F., Lusk, J., Davis, A., Ezeani, C., Perkins, M., & Pryer, B. (2022) · Issues in Mental Health Nursing · doi:10.1080/01612840.2022.2055243 · Cited by 7
“While numerous quality appraisal tools exist, the team decided to use the 2018 mixed methods appraisal tool (MMAT) for this integrative review (Hong et al., 2018).”
BackgroundMethodYoung, H. M. L., Yates, T. E., Dempsey, P. C., Herring, L. Y., Henson, J. J., Sargeant, J. A., Curtis, F., Sathanapally, H., HIGHTON, P. J., Hadjiconstantinou, M., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061104 · Cited by 5
Leroux, J., Johnston, N., Brown, A.-A., Mihic, A., DuBois, D., & Trudell, A. (2022) · INQUIRY: The Journal of Health Care Organization, Provision, and Financing · doi:10.1177/00469580221097427 · Cited by 4
Kalu, M. E., Okeke, C. O., Nwachukwu, E. C., Okoh, A. C., Akinrolie, O., Ezulike, J. C. D., Adandom, H. C., Onyeso, O. K., Egbumike, J., Olatunji, F. D., et al. (2022) · Journal of Global Health Economics and Policy · doi:10.52872/001c.36188 · Cited by 7
Lamb, C., Kennedy, M., Clark, A., Pituskin, E., Kirkwood, K. W., & Babenko‐Mould, Y. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-053880 · Cited by 4
Overen, C. K., Larsson, M., Hillestad, A. H., & Eriksen, S. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-063230 · Cited by 4
Knight, R. L., Sharp, C., Hallingberg, B., Mackintosh, K. A., & McNarry, M. A. (2022) · Children · doi:10.3390/children9111727 · Cited by 8
“Whilst 11 met 100% of the MMAT quality criteria [30–34,36–39,41], two articles only met 40% of the criteria [28,40], one met 60%, [42] and two met 80% [27,29].”
MethodInfluentialProjecting Malaria Incidence Based on Climate Change Modeling Approach: A Systematic Review
Baharom, M., Soffian, S. S. S., Peng, C. S., Baharudin, M. H., Mirza, U., Madrim, M. F., Jeffree, M. S., Rahim, S. S. S. A., & Hassan, M. R. (2022) · Open Access Macedonian Journal of Medical Sciences · doi:10.3889/oamjms.2022.10141 · Cited by 7
“Next, the researchers appraise the quality of the selected articles using the mixed methods appraisal tool (MMAT) Version 2018 [12] to ensure the quality of the articles for reviewing.”
MethodUtilising interactive technology for stroke patient education: a systematic review
Finch, E., Coleman, A., Cameron, A. E., & Moro, C. (2023) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2022.2161649 · Cited by 6
“The MMAT is designed to appraise the methodological quality of empirical studies with various designs and methodologies during a systematic review [21].”
MethodCetin‐Sahin, D., Cummings, G. G., Gore, G. C., Vedel, I., Karanofsky, M., Voyer, P., Gore, B., Lungu, O., & Wilchesky, M. (2023) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2022.12.025 · Cited by 5
Self-citationPetersen, B., Koshy-Chenthittayil, S., DeArmond, M., & Caromile, L. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0276089 · Cited by 6
“Since we are excluding gray literature, we are modifying the existing Mixed Methods Appraisal Tool (2018) [20] for our included studies.”
MethodTyldesley‐Marshall, N., Parr, J., Brown, A., Chen, Y., & Grove, A. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1124658 · Cited by 5
Ne, C. K. H., Suaini, N. H. A., Aung, W. T., Ong, K. G. S., Samuel, M., & Tham, E. H. (2023) · Journal of Allergy and Clinical Immunology Global · doi:10.1016/j.jacig.2023.100181 · Cited by 6
Burlingham, M., Maguire, L., Hibberd, L., Turville, N., Cowdell, F., & Bailey, E. (2023) · Public Health Nursing · doi:10.1111/phn.13259 · Cited by 12
Cottrell, K., & Chapman, H. M. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2023.2281601 · Cited by 5
“…Appraisal Skills Programme (CASP) [61] checklist was used for qualitative and quantitative studies and the Mixed Methods Appraisal Tool (MMAT) [62] for mixed method Charting the extracted data using a descriptive analytical method 5 Collating, summarising and reporting findings (presenting a…”
Summers, R. H., Lea, J., & East, L. (2024) · Contemporary Nurse · doi:10.1080/10376178.2024.2304712 · Cited by 7
“Included papers were evaluated and cross-checked using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
MethodAttreed, A., Morand, L. R., Pond, D. C., & Sturmberg, J. P. (2024) · Cureus · doi:10.7759/cureus.61703 · Cited by 5
“DBL compared to AD revealed significantly lower HRV markers (SDNN, RMSSD, pNN50, VLF, LF, TP, and LF-HRV), which may be helpful to distinguish the two types of dementia [3,4,37], and might allow an earlier diagnosis and potentially improved patient outcome.”
MethodResultvan Schaik, L. F., Engelhardt, E. G., Wilthagen, E. A., Steeghs, N., Coves, A. F., Joore, M. A., van Harten, W. H., & Retèl, V. P. (2024) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2024.104441 · Cited by 7
Dzinamarira, T., Kuupiel, D., & Mashamba-Thompson, T. P. (2020) · Systematic Reviews · doi:10.1186/s13643-019-1266-3 · Cited by 4
Gopalani, S. V., Sedani, A. E., Janitz, A. E., Clifton, S. C., Stoner, J. A., Peck, J. D., Comiford, A. L., Salvatore, A. L., & Campbell, J. E. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-035658 · Cited by 4
Valmari, E. G., Ghazinour, M., Nygren, U., & Gilenstam, K. (2021) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2021.1939415 · Cited by 7
“Therefore, all studies will be critically appraised with the Mixed Method Appraisal Tool (MMAT) [48].”
MethodShahali, S., Shariati, S., & Montazeri, A. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01496-2 · Cited by 4
In Search of A Contemporary Definition of Meaningful Engagement: A systematic literature review
Hobson, T. K., Fitzgerald, A., & Radford, K. (2021) · Asia Pacific Journal of Health Management · doi:10.24083/apjhm.v16i4.485 · Cited by 3
Interventions to Improve Outcomes of Grandchildren Raised by Grandparents: A Systematic Review
Xu, Y., Pace, S., McCarthy, L. P., Harrison, T. M., & Wang, Y. (2022) · Research on Social Work Practice · doi:10.1177/10497315221079352 · Cited by 8
“For the other two qualitative pre-experimental studies with one-shot case designs (Dannison & Smith, 2003; Dare, Marquis, Wenden, Gopi, & Coall, 2020), we employed a qualitative study assessment tool (7 items), which was derived from the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodBujold, M., Pluye, P., & Légaré, F. (2022) · Research Square · doi:10.21203/rs.3.rs-1185708/v1 · Cited by 4
Self-citationApproaches to improving mental health care for autistic people: a systematic reviewPreprint
Loizou, S., Pemovska, T., Stefanidou, T., Foye, U., Cooper, R. E., Kular, A., Greenburgh, A., Baldwin, H., Griffiths, J., Saunders, K. R. K., et al. (2023) · medRxiv · doi:10.1101/2023.03.10.23287101 · Cited by 5
Spooner, C., Vivat, B., White, N., & Stone, P. (2023) · JMIR Research Protocols · doi:10.2196/49774 · Cited by 3
“Methodological quality was assessed using the Mixed Methods Appraisal Tool [29].”
MethodHasbie, S. N. R., Assim, M. I. S. A., Taasim, S. I., & Anuar, F. N. (2023) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v13-i15/18799 · Cited by 5
Markowski, M., Kalocsányiová, E., & Essex, R. (2023) · Written Communication · doi:10.1177/07410883231207103 · Cited by 4
“The quality of retained studies was assessed using the Mixed Methods Appraisal Tool (MMAT), which is designed for the appraisal stage of systematic mixed studies reviews, that is, reviews that include qualitative, quantitative, and mixed methods studies (Hong et al., 2018).”
MethodRamadan, M., Hadfield, K., Ryan, M., Cai, P., Bosqui, T. J., & Nolan, A. (2024) · Arts & Health · doi:10.1080/17533015.2024.2395896 · Cited by 9
“The methodological and reporting quality of the included studies was conducted using The Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018).”
Physical and Mental Health Status of Tribal People in Odisha: A Systematic Literature Review
Behera, N., Pandey, K., & Shankar, S. (2024) · Global Social Welfare · doi:10.1007/s40609-024-00355-5 · Cited by 3
“MMAT assessment tool using the Hong et al. (2018) coding system examines the appropriateness of using mixed methods in empirical studies.”
Tiol, E. B. M., Franco, R. F., Spaziani, A. O., Silva, G. G., dos Santos, E. R., Brienze, V. M. S., Lima, A. R. D. A., Lopes, S. M. M., Faria, J. I. L., Werneck, A. L., et al. (2024) · Nursing Reports · doi:10.3390/nursrep14040248 · Cited by 4
Value Cocreation and Codestruction in Digital Health Services: Protocol for a Systematic Review
Laukka, E., Tuunanen, T., Jansson, M. M., Vanhanen, M., Hirvonen, N., Palukka, J., Vesinurm, M., & Torkki, P. M. (2024) · JMIR Research Protocols · doi:10.2196/63015 · Cited by 3
Kooijman, M.-A., van Leeuwen, G. L., Schuurmann, R. C. L., van Leeuwen, B. L., Munster, B. C. V., van der Wal‐Huisman, H., & de Vries, J. P. P. M. (2024) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2024.12.011 · Cited by 3
Bowen, A. M., Calder, R., Neale, J., Meynen, T., & Gilchrist, G. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241306362 · Cited by 6
“The MMAT includes criteria to assess the quality of empirical quantitative, qualitative, and mixed-methods studies.”
InfluentialPatmisari, E., Huang, Y., McLaren, C., Bhatia, P., Orr, M., Govindasamy, S., Hielscher, E., & McLaren, H. (2025) · Scandinavian Journal of Occupational Therapy · doi:10.1080/11038128.2025.2468421 · Cited by 4
“Quality appraisal of each study was completed by one researcher (EP) using the Mixed Methods Appraisal Tool (MMAT) [40], and re-checked by a second author (HM).”
Implementation of mHealth to support cancer diagnosis in Sub-Saharan Africa: A systematic review
Arendse, K. D., Baby, G. A., Maramba, T. T. M., Moodley, J., Walter, F. M., & Scott, S. E. (2025) · African Journal of Primary Health Care & Family Medicine · doi:10.4102/phcfm.v17i1.4683 · Cited by 3
“Two reviewers (K.D.A., G.A.B.) independently assessed each included study for the risk of bias using the Mixed Methods Appraisal Tool (MMAT), 28 an externally validated tool used to assess the risk of bias in mixed methods systematic reviews.”
The consequences of social camouflaging in autistic adults: A systematic review
Summerill, J., & Summers, S. J. (. (2025) · Research in Autism · doi:10.1016/j.reia.2025.202556 · Cited by 17
Alaboson, J., Coffey, L., Shrivastava, S., Ade-Alao, A. O., & Maguire, R. (2025) · Digital Health · doi:10.1177/20552076251326230 · Cited by 3
“JA and AA independently assessed the quality of included studies using the Mixed Methods Appraisal Tool (MMAT: 36 which categorizes studies based on methodology and critically assesses quality based on design).”
Wang, X., Tian, K., Hu, J., et al. (2025) · BMC Nephrology · doi:10.1186/s12882-025-04104-4 · Cited by 3
“Two researchers with expertise in kidney disease management and evidence-based medicine independently assessed study quality using the Mixed Meth-ods Appraisal Tool (MMAT) [30].”
Systematic Review of Effects of Medication Dispenser Use by Home-Dwelling Older Adults
Salmensuu, O., Hyttinen, V., Isotalo, J., Rijken, M., Linnosmaa, I., & Kaarakainen, M. (2025) · Nursing Research · doi:10.1097/nnr.0000000000000824 · Cited by 4
“For the initially identified five original studies, we used the “ Mixed Methods Ap-praisal Tool ” (MMAT; Hong et al., 2018).”
Health Professionals' Attitudes Towards Traditional Healing for Mental Illness: A Systematic Review
Molla, A., Usher, K. J., Rice, K., & Islam, M. S. (2025) · International Journal of Mental Health Nursing · doi:10.1111/inm.70043 · Cited by 5
“The Mixed Methods Appraisal Tool (MMAT) Version 2018 Checklist (Hong et al. 2018) was used to assess the qualities of the included articles.”
George, N. S., Bassey, A. E., Cruz, J. A. A., & Tambor, M. (2025) · Discover Public Health · doi:10.1186/s12982-025-00584-9 · Cited by 9
“MMAT enables the evaluation of methodological strength across five categories of studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies [26].”
Effectiveness of interventions on conscience: Findings of a systematic review
Lamb, C., Pouliopoulou, D. V., Kirkwood, K. W., et al. (2025) · Nursing Ethics · doi:10.1177/09697330251333386 · Cited by 3
“15,31,32 The MMAT is a critical appraisal tool to ascertain the quality of different study designs included in systematic reviews.”
Krishnamoorthy, Y., Govindan, D., Karunakaran, M., Parveen, M., Poul, A. J., Manohar, A., & Rajendran, G. (2025) · Simulation & Gaming · doi:10.1177/10468781251338382 · Cited by 7
Parsons, K., Payne, S. M., Bhaska, L., Wallace, J., & Holt, N. (2025) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2025.102867 · Cited by 3
Informal caregivers’ attitudes towards care robots: A mixed methods systematic review
Wang, J., Dong, X., Chen, S., Lu, X., Chen, X., Liu, H., Wen, S., Tang, S., Zhao, T., Hou, C., et al. (2025) · Ageing Research Reviews · doi:10.1016/j.arr.2025.102778 · Cited by 4
“MMAT is a quality appraisal instrument for mixed-method reviews, content validated and piloted across all methodologies.”
InfluentialInterventions to Improve the Sleep of Nurses: An Integrative Review
Owen, C., & Lach, H. W. (2025) · Western Journal of Nursing Research · doi:10.1177/01939459251341830 · Cited by 3
The impact of physician associates in primary care in the UK: a systematic review
Willacy, S., Ravindran, U., & Oyebode, O. O. (2025) · BJGP Open · doi:10.3399/bjgpo.2025.0026 · Cited by 3
Rawi, S. B. M., Rafi’i, M. R., Ibrahim, R. B., Ahmad, N., Sutan, R., & Devarajooh, C. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12944-x · Cited by 4
“Checklists from the Joanna Briggs Institute (JBI) were used for the evaluation of the observational or interventional studies [31], while the mixed-methods study was evaluated using the Mixed-Methods Appraisal Tool (MMAT), Version 2018 [32].”
Nichols, B., Aggar, C., & Huang, Y. (2025) · Journal of Clinical Nursing · doi:10.1111/jocn.17835 · Cited by 4
“…Methods Appraisal Tool (MMAT) was used to assess the methodological quality of qualitative, quantitative and mixed methods studies by focusing on five core criteria for each category: qualitative, randomised controlled, non-randomised, quantitative descriptive and mixed methods (Hong et al. 2018).”
Flynn, A. V., Bermingham, M., Caples, M., Curtin, M., Dalton, C., McLoughlin, G., O’Mahony, J., Vučen, S. R., & Saab, M. M. (2025) · Health Policy and Planning · doi:10.1093/heapol/czaf024 · Cited by 4
“The quality of the included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) (Hong et al. 2018), which includes questions tailored to different study designs including qualitative studies (seven items), randomized controlled trials (RCTs; seven items), non-RCTs (seven items), quantitative descriptive…”
Allen, S., McNeill, A., McDermott, C. J., et al. (2025) · Journal of Genetic Counseling · doi:10.1002/jgc4.70079 · Cited by 3
“The Mixed Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018 ) was used to assess the methodological quality of the studies.”
Self-citationCowan, S., Bridgeman, J., Rennie, S. C., et al. (2025) · ANZ Journal of Surgery · doi:10.1111/ans.70230 · Cited by 3
“Good MDT 1) [23].”
InfluentialThe Oral Health Disparity Among Refugee Children: A Systematic Review
Kandahari, N., Zada, F. M., & Farzal, Z. (2025) · Laryngoscope Investigative Otolaryngology · doi:10.1002/lio2.70194 · Cited by 3
Oh, O., Ulrich, C. M., Massimo, L. M., & Demiris, G. (2025) · Palliative Medicine · doi:10.1177/02692163251353013 · Cited by 9
“26 This tool includes two screening and five design-specific questions.”
Anazor, F. C., & Grech, M. (2025) · Journal of surgical education · doi:10.1016/j.jsurg.2025.103618 · Cited by 7
Mastwyk, S., Taylor, N. F., Lowe, A. C., Dalton, C., & Peiris, C. L. (2025) · Physiotherapy · doi:10.1016/j.physio.2025.101837 · Cited by 7
Damani, Q., Malik, G., & Bourke, S. L. (2025) · Journal of Professional Nursing · doi:10.1016/j.profnurs.2025.08.009 · Cited by 7
González, M., Yoo, P. Y., Noyek, S. E., MacLeod, B., Kee, M., Phoenix, M. P., Micsinszki, S. K., Knutson, M., Neilson, C. J., & Woodgate, R. L. (2025) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2025.1638513 · Cited by 3
“The Mixed Methods Appraisal Tool (MMAT) version 2018 (61) was used for this purpose given this version was speci fi cally developed for systematic reviews that integrate qualitative, quantitative, and mixed methods evidence (62, 63).”
Prevalence and determinants of e-cigarette use in South Asia: a systematic review
Jayasinghe, Y. A., Ratnapreya, S., Nawarathna, G. S., Peiris, P. M., Kanmodi, K. K., & Jayasinghe, R. D. (2025) · BMJ Global Health · doi:10.1136/bmjgh-2024-018363 · Cited by 4
“Thereby, a total of 17 articles were finally included in this systematic review (online supplemental table 4, figure 1).”
Social identity and pro-environmental behavior—a scoping review
Thye, T. (2025) · Environmental Research Letters · doi:10.1088/1748-9326/ae06ba · Cited by 6
“Based on the MMAT (Hong et al 2018), the studies were classified into the following study design types (see also figure 3): quantitative descriptive studies ( n = 57), quantitative randomized controlled trials ( n = 13), quantitative descriptive studies/quantitative randomized controlled trials ( n = 6), quantitative…”
InfluentialMcPhail, L., Smartt, C., Musyimi, C. W., Agorinya, J., Timothewos, S., Feyera, F., Tsigebrhan, R., Eshetu, T., Misganaw, E., Asher, L., et al. (2025) · The Lancet Psychiatry · doi:10.1016/s2215-0366(25)00206-8 · Cited by 3
“A mixed methods appraisal tool (MMAT) was used to assess the quality of any studies where the criteria could be applied (20).”
Training Programs for Thyroid Biopsy and Ablation: A Systematic Review
Bridgemohan, S., Ortiz, A., Garcia, G. P. R., Kolukula, S., Acosta, G. J., Samouce, C. C., Lampotang, S., Brito, J. P., & Ospina, N. S. (2025) · Endocrine Practice · doi:10.1016/j.eprac.2025.09.010 · Cited by 3
Mapping Nursing Roles in Long‐Term Care: A Systematic Review With Network Analysis
Park, K. H., & Kim, H. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.70242 · Cited by 3
“Two reviewers independently assessed the methodological quality of studies using the Mixed Methods Appraisal Tool (MMAT, 2018 version) (Hong et al. 2018).”
Online decision aids for contraceptive choices in women with chronic conditions: A systematic review
Gebresillassie, B. M., Doley, N., & Harris, M. L. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70555 · Cited by 3
Experiences of disabled students in online education: a systematic review
Lynch, S., Brunton, J., & Farrell, O. (2025) · Distance Education · doi:10.1080/01587919.2025.2562802 · Cited by 11
“The current study used Hong et al. (2018) Mixed Methods Appraisal Tool (MMAT) for quality appraisal, which is widely used in systematic reviews containing a mixture of qualitative, quantitative, and mixed methods studies (Clifford et al., 2018; Coyne et al., 2018; McGuinness & Guerin, 2024; Mikkelsen et al., 2019; Moo…”
InfluentialJans, C., Lucas, C., & Levett-Jones, T. L. (2025) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2025.101807 · Cited by 4
Interprofessional simulation co-debriefing practices: A systematic review
Jacob, E. R., Parrish, T., Duffy, C., et al. (2025) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2025.101816 · Cited by 3
Onyejesi, C. R., James, T., & Kharicha, K. (2025) · Journal of Health Services Research & Policy · doi:10.1177/13558196251384845 · Cited by 4
Anderson, M. S., Baldwin, S., Rayment‐Jones, H., & Challacombe, F. L. (2025) · Midwifery · doi:10.1016/j.midw.2025.104630 · Cited by 4
Accounting Controls, Governance, and Anti-Corruption in Ghana’s Public Sector
Bediako, R., Ocansey, E. O. N. D., & Oppong, D. (2025) · African Journal of Accounting and Financial Research · doi:10.52589/ajafr-bp8qzkvc · Cited by 3
Elevating patient safety education in undergraduate nursing: An integrative review beyond trends
Villalba, C. C., Sigua, R. N. T., & Moya, J. L. M. (2025) · International Journal of Nursing Sciences · doi:10.1016/j.ijnss.2025.10.009 · Cited by 5
“To complement this and assess the methodological rigour of the diverse study designs, version 2018 of the Mixed Methods Appraisal Tool (MMAT) was employed [30].”
The psychological implications of forced adoption: A scoping review
O'Connor, M., Moroney, D., Muldoon, O. T., et al. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105773 · Cited by 3
Antenatal care interventions in Afghanistan from 2000 to 2024: a rapid realist review
Naseri, S., Durand, M., Arifi, F., Manochehr, M. H., Safi, N., Hadad, A. S., Rahmanzai, A. J., Yaqubi, G. S., & Ridde, V. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-102454 · Cited by 4
“The remaining 63 articles underwent full-text assessment for eligibility, resulting in the inclusion of 25 studies in the final review (figure 1).”
Coyle, N., Crowley, A., McNicholl, A., Gallagher, P., & Dunne, S. K. (2025) · Journal of Cancer Survivorship · doi:10.1007/s11764-025-01900-7 · Cited by 4
Exploring nutritional education in undergraduate nursing students: A scoping review
Perez-Garcia, I. N., Andrade-Gómez, E., Arias-Colinas, M., & Pardavila‐Belio, M. I. (2025) · Nurse Education in Practice · doi:10.1016/j.nepr.2025.104623 · Cited by 6
Blatgé, H., Paul, L., & van Wijck, F. (2025) · Clinical Rehabilitation · doi:10.1177/02692155251385222 · Cited by 3
Anson, K., & Patel, S. E. (2025) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2025.11.008 · Cited by 5
The Impact of Digital Technology on the Physical Health of Older Workers: Scoping Review
Spijker, J., Barlın, H., Dritsaki, M., Gu, Y., Kļaviņa, A., Yaylagül, N. K., Kulla, G., Mercan, M. A., Orhun, E., Ševčíková, A., et al. (2025) · JMIR Aging · doi:10.2196/78406 · Cited by 4
“We assessed study quality using the Mixed Methods Appraisal Tool (MMAT) [50], which evaluates five study design categories: (1) qualitative studies, (2) quantitative randomized controlled trials, (3) quantitative nonrandomized studies, (4) quantitative descriptive studies, and (5) mixed methods…”
Psychological interventions for dancers’ mental health: a systematic review
Zhang, Y., Noh, Y., & Liu, S. (2025) · BMC Psychology · doi:10.1186/s40359-025-03724-7 · Cited by 4
“We used the MMAT [52] to assess the quality of included studies, as this assessment tool includes criteria for five research design types (i.e., qualitative, randomized controlled, nonrandomized, quantitative descriptive, and mixed methods).”
InfluentialCarter, E., Douglas, C. C., & Saxton, J. (2025) · Health Psychology Review · doi:10.1080/17437199.2025.2597312 · Cited by 5
“The quality of the included studies was assessed using the mixed methods appraisal tool (MMAT; Hong et al., 2018), which has been used in previous GE reviews to assess a variety of study methods using a single tool (Mmako et al., 2020; Thomas et al., 2022).”
Busshart, L., Petrovic, M., Amin, R., & Hegerl, U. (2026) · JMIR mhealth and uhealth · doi:10.2196/70840 · Cited by 4
“While this decision was made to enhance clarity and focus, it inevitably limited the breadth of the review, as noted in other recent scoping and mixed methods reviews as well [13,45].”
Possner, M., Schwitz, F., Zach, N., Karrer, T., Tam, J., Oechslin, E., Muhll, I. V., Wolofsky, K. T., Birnie, D., Beauchesne, L., et al. (2025) · CJC Pediatric and Congenital Heart Disease · doi:10.1016/j.cjcpc.2025.09.002 · Cited by 3
“Using the MMAT, most quantitative descriptive, qualitative, and nonrandomized studies met the majority of criteria, indicating acceptable methodological quality.”
Fahey, J. T., Bramah, C., Barber, R., Williams, J. R., McMahon, J. J., Comfort, P., & Ripley, N. J. (2025) · Journal of Sports Sciences · doi:10.1080/02640414.2025.2606621 · Cited by 3
“The methodological quality of articles was assessed using a mixed methods appraisal tool (Hong et al., 2018).”
Kennedy, A., Mockler, D., Walsh, C., McQuillan, R., Ryan, K., Guinan, E. M., & Foley, G. (2025) · BMJ Supportive & Palliative Care · doi:10.1136/spcare-2025-005783 · Cited by 4
“The Mixed Methods Appraisal Tool (MMAT) 22 was used to assess the quality of the included studies.”
Gheri, F., Ceccanti, L., Donnarumma, F., & Petta, D. (2025) · Teaching and learning in nursing · doi:10.1016/j.teln.2025.11.036 · Cited by 3
Understanding Mental Fatigue Across the Men’s Basketball Season: A Mixed-Methods Study
Sansone, P., Ciaccioni, S., Polverari, D., García-Calvo, T., & Russell, S. (2026) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2025-0489 · Cited by 6
Effect of Digital Health Interventions on College Students’ Lifestyle Behaviors: Systematic Review
Zhou, Q., Jiang, J., Yin, Z., & Fan, R. (2026) · Journal of Medical Internet Research · doi:10.2196/82192 · Cited by 7
“…MMAT is designed to evaluate 5 categories of study designs: qualitative research (QR), quantitative randomized controlled trials (QRCTs), quantitative nonrandomized studies (QNRSs), quantitative descriptive studies (QDSs), and mixed methods studies (MMSs), each comprising 5 appraisal criteria [28].”
Rixon, A., Phelan, A., Ekberg, S., Senyard, J. M., Mashhady, A., Shaw, A., & Birdthistle, N. (2026) · Nurse Education in Practice · doi:10.1016/j.nepr.2026.104716 · Cited by 6
Bainazarova, N. T., Zhumabayeva, К., Yermukhanova, G., Hajiyeva, Y. H., Abdykerimova, K., Orakbay, L. Z., Zhumageldiyeva, F., Karibayeva, I. K., & Jainakbayev, N. T. (2026) · Frontiers in Dental Medicine · doi:10.3389/fdmed.2025.1726952 · Cited by 5
Peer Support for Women Who Have Experienced Interpersonal Trauma or Abuse: An Integrative Review
Thomson, G. M., Cull, J., Kuroski, J. A., Kuipers, Y., Leinweber, J., Karlsdóttir, S. I., & Burns, L. (2026) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.70179 · Cited by 4
A Systematic Literature Review of Servant Leadership in Education
Hossain, M. F., Uddin, M. S., Rana, M. S., & Siddique, M. N.-E.-A. (2026) · Journal of Ethnic and Cultural Studies · doi:10.29333/ejecs/2562 · Cited by 11
“To evaluate the methodological rigor of the studies included in this review, we applied the Mixed Methods Appraisal Tool (MMAT) using its quantitative evaluation criteria (Hong et al., 2018).”
Mindfulness-based interventions for gambling disorder: A systematic review
Tannous, A., Demetrovics, Z., Chhabra, B., Logemann, H. N. A., Czakó, A., Griffiths, M. D., & Szabó, A. (2026) · Journal of Behavioral Addictions · doi:10.1556/2006.2025.00100 · Cited by 5
“The assessment involved fi ve speci fi c questions tailored to each study ’ s research design, with possible answers being ‘ Yes ’ , ‘ No ’ , or ‘ Uncertain ’ (Hong et al., 2018).”
InfluentialLiao, G., Liu, Z., & Wan, H. (2026) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2026.100506 · Cited by 5
“From: Hong et al. (2018).”
InfluentialMubarak, F. (2026) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-08-2025-0212 · Cited by 10
Mental Imagery of the Self in Body Dysmorphic Disorder: A Mixed‐Methods Systematic Review
Hill, S., Hotton, M., Wallace, M., Veale, D. M., & Lau‐Zhu, A. (2026) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.70229 · Cited by 4
“As the MMAT recommends against overall quality ratings, tabulated ratings for each MMAT criterion are presented, with an accompanying description of trends (Section 3.1.1) as recommended by Hong et al. (2018).”
InfluentialDoctor–patient communication in epilepsy care: An integrative review
Tiwari, H. K., Singh, M. B., Raman, U., Acharya, S., Sharma, M., Parihar, J., Haldar, P., Bhatia, R., Singh, G., Maulik, P. K., et al. (2026) · Epileptic Disorders · doi:10.1002/epd2.70203 · Cited by 3
Sohrabi, H., Ghavam, S. M., Azadnajafabad, S., & Rezaei, N. (2026) · Cancer Medicine · doi:10.1002/cam4.71637 · Cited by 4
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.