The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 701–800 of 4,578
Hinneh, T. K., Akyirem, S., Bossman, I. F., Lambongang, V., Ofori-Aning, P., Ogungbe, O., & Commodore-Mensah, Y. Y. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001931 · Cited by 39
O'Connell, M., Barry, J., Hartigan, I., Cornally, N., & Saab, M. M. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17114 · Cited by 28
Access, acceptance and adherence to cancer prehabilitation: a mixed-methods systematic review
Watts, T. E., Courtier, N., Fry, S., Gale, N., Gillen, E., McCutchan, G. M., Patil, M., Rees, T., Roche, D., Wheelwright, S. J., et al. (2024) · Journal of Cancer Survivorship · doi:10.1007/s11764-024-01605-3 · Cited by 37
“%) of publications were about research of the highest quality, defined as having met 100% of the MMAT criteria [23].”
BackgroundMethodInfluentialJeng, A. (2024) · Social Psychology of Education · doi:10.1007/s11218-024-09920-4 · Cited by 33
“…with recommendations outlined by the creators of the MMAT, the methodological quality of each source was not reduced to a numeric score because doing so would have obscured nuances in where specific studies experienced limitations in methodological quality (Hong, Fàbregues, et al., 2018).”
Leese, C. J., Gupte, D., Christogianni, A., Higgins, C., Adair, P., Dall, P., Cameron, P. A., Smith, B. H., & Colvin, L. (2024) · Pain · doi:10.1097/j.pain.0000000000003314 · Cited by 30
Draganidis, A., Fernando, A. N., West, M. L., & Sharp, G. (2024) · Social Science & Medicine · doi:10.1016/j.socscimed.2024.117231 · Cited by 46
Ng’oda, M., Gatheru, P. M., Oyeyemi, O. T., Busienei, P. J., Karugu, C. H., Mugo, S., Okoth, L., Nampijja, M., Kiwuwa-Muyingo, S., Wado, Y. D., et al. (2024) · PLOS Global Public Health · doi:10.1371/journal.pgph.0003314 · Cited by 28
The use of leaderboards in education: A systematic review of empirical evidence in higher education
Li, C., Liang, L., Fryer, L. K., & Shum, A. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13077 · Cited by 55
A rapid review of the evidence for online interventions for bereavement support
Finucane, A., Canny, A., Mair, A. P. A., Harrop, E., Selman, L. E., Swash, B., Wakefield, D., & Gillanders, D. T. (2024) · Palliative Medicine · doi:10.1177/02692163241285101 · Cited by 34
“We used the AMSTAR (A MeaSurement Tool to Assess Systematic Reviews) Version 2 27 to assess the quality of systematic reviews, and the MMAT (Mixed Methods Appraisal Tool) 28 to appraise all other studies.”
Chung, J., & Jung, H. (2024) · BMC Nursing · doi:10.1186/s12912-024-02531-6 · Cited by 42
“For mixed-methods studies, we used the Mixed-Methods Appraisal Tool, Version 2018 [19], which consists of 17 questions including screening, qualitative component, quantitative component, and mixed-methods component questions.”
Dating apps and their relationship with body image, mental health and wellbeing: A systematic review
Bowman, Z., Drummond, M., Church, J., Kay, J., & Petersen, J. M. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108515 · Cited by 43
González, I. T., Lucena, F. J. H., Rodríguez, J. M. R., & Menéndez, A. M. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1480908 · Cited by 51
Abdulazeem, H. M., Meckawy, R., Schwarz, S., Novillo-Ortiz, D., & Klug, S. J. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105945 · Cited by 40
Quality of life and falls in elderly people: a mixed methods study
da Silva, T. L., da Motta, V. V., Garcia, W. J., et al. (2021) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2020-0400 · Cited by 24
“This is a mixed methods study of convergent parallel type(15), which used concomitant triangulation from two approaches (descriptive sectional study and structural approach of TSR)(1113,16-17), structured according to the Mixed Methods Appraisal Tool (MMAT) protocol, version 2018(18).”
MethodDeys, L. J., Wilson, V., & Meedya, S. (2021) · Midwifery · doi:10.1016/j.midw.2021.103063 · Cited by 33
Recommendations for navigating the experiences of moral distress: A scoping review
Deschenes, S., Tate, K., Scott, S. D., & Kunyk, D. (2021) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104035 · Cited by 28
Versluis, A., Schnoor, K., Chavannes, N. H., & Talboom-Kamp, E. P. W. A. (2021) · Journal of Medical Internet Research · doi:10.2196/29303 · Cited by 23
“1 | e29303 | p. 10https://www.jmir.org/2022/1/e29303 (page number not for citation purposes) XSL•FO RenderX MMAT scoresbMMAT quality criteriaaIncluded studies 54321 3+−+−+Rüütel et al [45] 4++/−+++Spielberg et al [46] 3++/−+++/−Zhong et al [47] 4+−+++Grandahl et al [48] 4+−+++Dulai et al [49] 4+−+++Talboom-Kamp et al…”
MethodInfluentialBrunner, M., Rietdijk, R., & Togher, L. (2022) · Journal of Medical Internet Research · doi:10.2196/35595 · Cited by 25
“A review protocol was written before commencing this study (Multimedia Appendix 1 [55-65]), with systematic searching initially conducted in March 2020 (by the first author [MB] and a student intern) and a secondary search conducted in November 2020 (by the first author, MB).”
MethodSaatchi, B., Agbayani, C. G., Clancy, S., & Fortier, M. A. (2022) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12851 · Cited by 30
Sahoo, K. C., Dubey, S., Dash, G. C., Sahoo, R. K., Sahay, M. R., Negi, S., Mahapatra, P., Bhattacharya, D., Sahoo, B., Pani, S. P., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191911845 · Cited by 26
“Authors (K.C.S. and M.R.S.) assessed the quality of the selected studies, which were assessed using the Mixed Methods Appraisal Tool [20] (Appendix 5).”
MethodDuan, L., Liu, C., Wang, D., Lin, R., Qian, P., Zhang, X., & Liu, C. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.985188 · Cited by 37
“The risk of bias of the included studies was assessed using the JBI checklist for quantitative studies (33), the Critical Appraisals Skills Programme (CASP) checklist for qualitative studies (34), and the Mixed Methods Appraisal Tool (MMAT) (35) for mixed methods studies, respectively.”
MethodSystematic Review of Smoking Cessation Interventions for Smokers Diagnosed with Cancer
Frazer, K., Bhardwaj, N., Fox, P., Stokes, D., Niranjan, V. R., Quinn, S., Kelleher, C. C., & Fitzpatrick, P. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192417010 · Cited by 24
“Hong, Q.N.; Fàbregues, S.; Bartlett, G.; Boardman, F.; Cargo, M.; Dagenais, P.; Gagnon, M.-P.; Griffiths, F.; Nicolau, B.; O’Cathain, A.; et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers.”
MethodInfluentialPahri, S. D. R., Er, A. C., & Cheah, W. Y. (2023) · Sustainability · doi:10.3390/su15043123 · Cited by 52
“Two reviewers implemented the Mixed-Method Appraisal Tool (MMAT) developed by Hong et al. [30] to evaluate the quality of the articles.”
Haghighi, M. H. H., & Alipour, J. (2023) · Burns · doi:10.1016/j.burns.2023.07.001 · Cited by 26
Govindaraj, R., Agar, M. R., Currow, D. C., & Luckett, T. (2023) · Journal of Medical Internet Research · doi:10.2196/48483 · Cited by 32
“To assess the rigor of the included studies, a validated assessment tool, the Mixed Methods Appraisal Tool version 2018 [27], was used, which assesses the trustworthiness at the data source level.”
Martin, S., Richards, C., Keogh, S. J., & Ward, A. (2024) · Teaching and learning in nursing · doi:10.1016/j.teln.2023.11.008 · Cited by 29
Lin, H., Ye, M., Chan, S. W., Zhu, J., & He, H. G. (2020) · Gynecologic Oncology · doi:10.1016/j.ygyno.2020.04.690 · Cited by 28
The Need to Rigorously Develop Common Quality Guidelines for Reporting Mixed Methods Research
Guetterman, T. C., Molina-Azorín, J. F., & Fàbregues, S. (2022) · Journal of Mixed Methods Research · doi:10.1177/15586898221143561 · Cited by 30
Self-citationRamachandran, S., Chang, H.-J., Worthington, C., Kushniruk, A., Ibáñez-Carrasco, F., Davies, H., McKee, G., Brown, A., Gilbert, M., & Iyamu, I. (2024) · JMIR Public Health and Surveillance · doi:10.2196/52798 · Cited by 23
Jackman, P. C., Lane, A., Wells, N., Kirby, K., & Bird, M. D. (2023) · International Journal of Sport and Exercise Psychology · doi:10.1080/1612197x.2023.2257214 · Cited by 25
“As quantitative, qualitative, and mixed-method studies were eligible, we used the mixed methods appraisal tool (MMAT; Hong et al., 2018).”
InfluentialLived experience of fetal alcohol spectrum disorder: A qualitative scoping review
Wilson, H., McCormack, J. C., Singh, N., & Chu, J. T. W. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15761 · Cited by 23
“18,19 The first author applied the quality assessment tool to all articles, with a 10% random sample reviewed by another author (NS); any disagreements were discussed among all reviewers.”
MethodInterventions for improving mental health in athletes: a scoping review
Ekelund, R., Holmström, S., Gustafsson, H., Ivarsson, A., Lundqvist, C., & Stenling, A. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2023.2258383 · Cited by 41
“Studies with a group-based design were critically appraised with the Mixed Methods Appraisal Tool (MMAT) Version 2018 (Hong et al., 2018).”
MethodHeng, P. P., Yusoff, H. M., & Hod, R. (2024) · PLoS ONE · doi:10.1371/journal.pone.0287892 · Cited by 32
Interventions for left-behind children in Mainland China: A scoping review
Jia, Z., & Wu, Q. (2024) · Children and Youth Services Review · doi:10.1016/j.childyouth.2024.107933 · Cited by 24
Decent and Precarious Work Among Nursing and Care Workers: A Mixed‐Method Systematic Review
Hult, M., Ring, M. A., Siranko, H., & Kangasniemi, M. K. (2024) · Journal of Advanced Nursing · doi:10.1111/jan.16572 · Cited by 26
“We employed the Mixed Methods Appraisal Tool (MMAT) checklist for assessing the quality of all included studies (Hong et al. 2018).”
Yin, H., Zhang, J., Lian, M., & Zhang, Y. (2025) · BMC Public Health · doi:10.1186/s12889-025-22829-7 · Cited by 41
Peterson, J. M., & Jensen, M. T. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1531481 · Cited by 42
“In addition to data extraction, a quality and relevance appraisal for each study was carried out using the seven questions provided on the Mixed Methods Appraisal Tool (MMAT) within EPPI reviewer (Hong et al., 2018; Thomas et al., 2023).”
Scheffers, F., Moonen, X., & van Vugt, E. (2020) · Journal of Intellectual Disabilities · doi:10.1177/1744629520961942 · Cited by 32
“To evaluate the quality of the studies the Mixed Method Appraisal Tool (MMAT) (Hong et al., 2018) was used to describe the methodological quality for three domains: qualitative, quantitative and mixed-method studies.”
MethodSarigiovannis, P., Jowett, S. M., Saunders, B., & Bishop, A. (2020) · Physiotherapy · doi:10.1016/j.physio.2020.10.002 · Cited by 23
A Scoping Review of the Whole School, Whole Community, Whole Child Model
Willgerodt, M. A., Walsh, E., & Maloy, C. (2020) · The Journal of School Nursing · doi:10.1177/1059840520974346 · Cited by 27
“The MMAT is used to describe the quality of empirical studies.”
MethodInfluentialGasteiger, N., van der Veer, S. N., Wilson, P. M., & Dowding, D. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-050033 · Cited by 37
A Systematic Review on Adaptation Practices in Aquaculture towards Climate Change Impacts
Samah, A. A., Shaffril, H. A. M., Fadzil, M. F. B., Ahmad, N. B., & Idris, K. (2021) · Sustainability · doi:10.3390/su132011410 · Cited by 29
Kirby, K., Voss, S. E., Bird, E. L., & Benger, J. R. (2021) · Resuscitation Plus · doi:10.1016/j.resplu.2021.100173 · Cited by 23
“Originally developed in 2006,(13) the tool was revised in 2011(18) The current version was further revised following a Delphi study, interviews with MMAT users and a literature review of critical appraisal tools.(12) Each paper was scored using the MMAT.”
MethodInfluentialFrances, S. M., Velikova, G., Klein, M., Short, S. C., Murray, L. J., Wright, J. M., & Boele, F. W. (2021) · Neuro-Oncology Practice · doi:10.1093/nop/npab062 · Cited by 31
Current state of preconception care in sub-Saharan Africa: A systematic scoping review
Ukoha, W. C., Mtshali, G. N., & Adepeju, L. (2022) · African Journal of Primary Health Care & Family Medicine · doi:10.4102/phcfm.v14i1.3096 · Cited by 29
Wernli, K., Smith, A. J., Coll, F., Campbell, A., Kent, P. M., & O’Sullivan, P. (2022) · European Journal of Pain · doi:10.1002/ejp.2022 · Cited by 27
Schmitt, N. N., Striebich, S., Meyer, G., Berg, A., & Ayerle, G. M. (2022) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-022-05014-1 · Cited by 23
“Because we combined qualitative, quantitative and mixed-methods studies, we used the Mixed Methods Appraisal Tool (MMAT) to assess the quality of the studies [42].”
MethodDisentangling Rates, Risk, and Drivers of Suicide in the Construction Industry
Tyler, S., Hunkin, H., Pusey, K., Gunn, K. M., Clifford, B., McIntyre, H., & Procter, N. G. (2022) · Crisis · doi:10.1027/0227-5910/a000885 · Cited by 23
“Studies were independently assessed using the Mixed Methods Appraisal Tool, with ST reviewing all studies and KP cross-checking 40% (randomly selected) of studies for inter-rater reliability, with 100% concordance (the results shown in table in ESM 3; Hong et al., 2018).”
MethodJameson, J., Barnard, J., Rumyantseva, N. L., Essex, R., & Gkinopoulos, T. (2022) · Studies in Higher Education · doi:10.1080/03075079.2022.2145278 · Cited by 32
“February 2022) using the Mixed Methods Appraisal Tool (MMAT), Version 2018 (Hong et al. 2018).”
MethodSmith, E., Gullick, J. G., Perez, D., & Einboden, R. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16600 · Cited by 31
Vaccine hesitancy in cancer patients: A rapid review
Butow, P. N., Shaw, J., Bartley, N., Milch, V., Sathiaraj, R., Turnbull, S., & Vartanian, C. D. (2023) · Patient Education and Counseling · doi:10.1016/j.pec.2023.107680 · Cited by 28
“The Mixed-Methods Quality Appraisal Tool (MMAT) [12] was used to assess for potential bias, as it is applicable to all study types.”
MethodInfluentialGauly, J., Court, R. A., Currie, G., Seers, K., Clarke, A., Metcalfe, A. J., Wilson, A. C., Hazell, M., & Grove, A. (2023) · Implementation Science · doi:10.1186/s13012-023-01274-3 · Cited by 29
“These high profile 1-day courses aim to optimise and enhance the performance of individual surgeons [29], yet the evidence for their effectiveness is limited to attendees or peers self-reports of changes in leadership skills [44, 57].”
BackgroundMethodSun, X., Zhang, J., Wang, Y., Zhang, X., Li, S., Qu, Z., & Zhang, H. (2023) · Brain and Behavior · doi:10.1002/brb3.3108 · Cited by 38
“MMAT can assess the methodological quality of five types of studies, namely, qualitative studies, randomized controlled trials, nonrandomized studies, quantitative descriptive studies, and mixed studies (Hong et al., 2018).”
Manship, S., Hatzidimitriadou, E., Moore, J., Stein, M., Towse, D., & Smith, R. (2023) · Assistive Technology · doi:10.1080/10400435.2023.2219294 · Cited by 27
“%) out of a maximum of 10 using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodIntelligence Quotient (IQ) in school-aged preterm infants: A systematic review
Lacalle, L., Martínez‐Shaw, M. L., Almagro, Y. M., & Sánchez‐Sandoval, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1216825 · Cited by 33
“The quality of the evidence of the selected studies was verified with the Mixed Methods Appraisal Tool [MMAT; (Hong et al., 2018)].”
MethodLee, M. F., Bolton, K., Madsen, J., & Burke, K. J. (2023) · Journal of Reproductive and Infant Psychology · doi:10.1080/02646838.2023.2252453 · Cited by 35
“The MMAT assesses the methodological quality of qualitative, quantitative and mixed-methods research papers (Hong et al., 2018).”
Relationship between staff and quality of care in care homes: StaRQ mixed methods study
Spilsbury, K., Charlwood, A., Thompson, C. A., Haunch, K. J., Valizade, D., Devi, R., Jackson, C., Alldred, D. P., Arthur, A. J., Brown, L., et al. (2024) · Health and Social Care Delivery Research · doi:10.3310/gwtt8143 · Cited by 42
Limited access to care for persons with breast cancer in Africa: A systematic review
Salisu, W. J., Mirlashari, J., Varaei, S., & Seylani, K. (2020) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2020.101867 · Cited by 24
Approaches to improving mental healthcare for autistic people: Systematic review
Loizou, S., Pemovska, T., Stefanidou, T., Foye, U., Cooper, R., Kular, A., Greenburgh, A., Baldwin, H., Griffiths, J., Saunders, K. R. K., et al. (2024) · BJPsych Open · doi:10.1192/bjo.2024.707 · Cited by 22
“All MMAT ratings can be seen in Supplementary Table 7.”
Lim, R. B. T., Hoe, K. W. B., & Zheng, H. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.938224 · Cited by 33
Xie, E. B., Jung, J. W., Kaur, J., Benzies, K., Tomfohr‐Madsen, L. M., & Keys, E. (2023) · Journal of Medical Internet Research · doi:10.2196/43219 · Cited by 30
Facilitators and barriers to the implementation of surgical safety checklist: an integrative review
Lim, P. J. H., Chen, L., Siow, S., & Lim, S. H. (2023) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzad086 · Cited by 37
Bott, D., Subramanian, A., Edgar, D. F., Lawrenson, J. G., & Campbell, P. (2023) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13245 · Cited by 29
Tobin, M., Carney, S., & Rogers, E. M. (2023) · Infant Mental Health Journal · doi:10.1002/imhj.22091 · Cited by 23
Challenges Faced by Medical Rescue Teams During Disaster Response: A Systematic Review Study
Kuday, A. D., Özcan, T., Çalışkan, C., et al. (2023) · Disaster Medicine and Public Health Preparedness · doi:10.1017/dmp.2023.217 · Cited by 22
Contextualizing institutional capacity in water governance framework: a literature review
Ariffin, R. N. R., Sawon, S., Rahman, N. H. A., Hanafi, H. B., & Zahari, R. K. (2023) · Water Policy · doi:10.2166/wp.2023.074 · Cited by 37
Schröder, J., Riiser, K., & Holmen, H. (2024) · BMC Health Services Research · doi:10.1186/s12913-023-10495-7 · Cited by 23
“MMAT is a critical appraisal tool for five different study designs in qualitative, quantitative, and mixed methods studies [16].”
MethodAllory, E., Scheer, J., Andrade, V. D., Garlantézec, R., & Gagnayre, R. (2024) · BMC Family Practice · doi:10.1186/s12875-024-02262-2 · Cited by 24
“First, the quality of the included studies was evaluated using the MMAT [35].”
MethodValue-Based Integrated Care: A Systematic Literature Review
van Hoorn, E. S., Ye, L., van Leeuwen, N., Raat, H., & Lingsma, H. F. (2024) · International Journal of Health Policy and Management · doi:10.34172/ijhpm.2024.8038 · Cited by 34
“22 This review, therefore, only assessed the methodological quality of the empirical studies.”
BackgroundMethodZhang, Z. X., Lai, J., Shen, L., & Krishna, L. (2024) · British Journal of Sports Medicine · doi:10.1136/bjsports-2023-107717 · Cited by 35
“37 Disagreements regarding appraisal were resolved through discussion between both reviewers and with a third reviewer (LK) when consensus could not be reached.”
BackgroundGross, A., Olson, K. A., Pool, J. J. M., Basson, A., Clewley, D. J., Dice, J. L., & Milne, N. (2024) · Journal of Manual & Manipulative Therapy · doi:10.1080/10669817.2024.2332026 · Cited by 24
“The level of evidence statement from the scoping review considered quality and quantity of evidence findings from systematic reviews and high-quality studies (≥5/7 on critical appraisal on the Mixed Methods Appraisal Tool) [20].”
BackgroundFeig, V. R., Zhang, S., Patel, A., Santos, B., Kang, Z., Wasan, S. K., Beloqui, A., & Traverso, G. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2024.102850 · Cited by 26
Andreoli, L., Peeters, H., Steen, K. V., & Dierickx, K. (2024) · European Journal of Human Genetics · doi:10.1038/s41431-024-01747-z · Cited by 26
“The fi rst author critically evaluated the included studies using the Mixed Methods Appraisal Tool (MMAT) [28] (table available in Supplement 3).”
Conversational AI in Tourism: A systematic literature review using TCM and ADO framework
Prasanna, A., Pushparaj, P., & Kushwaha, B. P. (2025) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2025.101310 · Cited by 38
Big data in financial risk management: evidence, advances, and open questions: a systematic review
Theodorakopoulos, L., Theodoropoulou, A., & Bakalis, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1658375 · Cited by 22
Savolainen, I., Ylinen, L., Grönroos, R., & Oksanen, A. (2025) · Digital Business · doi:10.1016/j.digbus.2025.100162 · Cited by 62
Uhm, J., & Choi, M. (2020) · Healthcare · doi:10.3390/healthcare8040506 · Cited by 23
“independently used the Mixed Method Appraisal Tool (MMAT), version 2018 [22], to appraise the”
MethodKlein, P., Fairweather‐Schmidt, A. K., Lawn, S., Stallman, H. M., & Cammell, P. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01580-1 · Cited by 30
Birnie, K. A., Pavlova, M., Neville, A., Noel, M. E., Jordán, I., Jordan, E., Marianayagam, J., Stinson, J., Lorenzetti, D., Faulkner, V., et al. (2021) · Pain · doi:10.1097/j.pain.0000000000002339 · Cited by 23
“All records were independently assessed for quality using the Mixed Methods Appraisal Tool (MMAT-v2018) as applicable and appropriate across study types, including qualitative, quantitative RCTs, quantitative nonrandomized, quantitative descriptive, and mixed methods studies.(26,39) The MMAT provides 2 methodological…”
MethodAlcohol and substance use prevention in Africa: systematic scoping review
Mupara, L. M., Tapera, R., Selemogwe-Matsetse, M., Kehumile, J. T., Gaogane, L., Tsholofelo, E., & Murambiwa, P. (2021) · Journal of Substance Use · doi:10.1080/14659891.2021.1941356 · Cited by 32
“Therefore, the quality of the studies was determined through study appraisal using the mixed-method appraisal tool (MMAT) – Version 2018 (Nha Hong et al., 2018).”
MethodAskelin, B., Hind, A., & Paterson, C. C. (2021) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2021.102032 · Cited by 24
Professional values in student nurse education: An integrative literature review
Antoniou, C., Clifton, R., & Wilson, V. (2022) · Nursing Ethics · doi:10.1177/09697330221085772 · Cited by 27
Tidmarsh, G., Thompson, J. L., Quinton, M. L., & Cumming, J. (2022) · Journal of Youth Development · doi:10.5195/jyd.2022.1156 · Cited by 32
“…JBI quality assessment for qualitative studies (Lockwood et al., 2015), JBI quality assessment for analytical cross-sectional studies (Moola et al., 2020), JBI quality assessment for randomised control trials (Tufanaru et al., 2020), and the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodHohmann, L. A., & Garza, K. B. (2022) · Pharmacy · doi:10.3390/pharmacy10040085 · Cited by 28
“Risk of bias (ROB) was assessed within each study using the mixed methods appraisal tool (MMAT) version 2018 [26].”
Bowdler, S., Nielsen, W., Moroney, T. L., & Meedya, S. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103423 · Cited by 31
Muls, A., Georgopoulou, S., Hainsworth, E., Hartley, B., O'Gara, G., Stapleton, S., & Cruickshank, S. (2022) · Seminars in Oncology · doi:10.1053/j.seminoncol.2022.08.001 · Cited by 22
Zahroh, R. I., Hazfiarini, A., Eddy, K. E., Vogel, J. P., Tunçalp, Ö., Minckas, N., Althabe, F., Oladapo, O. T., & Bohren, M. A. (2022) · PLoS Medicine · doi:10.1371/journal.pmed.1004074 · Cited by 29
“Three reviewers (RIZ, KEE, and AH) assessed methodological limitations for each study using an adapted Mixed Methods Appraisal Tool (MMAT) (use of critical appraisal tool changed from protocol version) [30].”
Young-onset dementia: A systematic review of the psychological and social impact on relatives
Wiggins, M., McEwen, A., & Sexton, A. (2022) · Patient Education and Counseling · doi:10.1016/j.pec.2022.107585 · Cited by 32
Storytelling in Pregnancy and Childbirth: An Integrative Review of the Literature
Mahdavi, Z., Amiri‐Farahani, L., & Pezaro, S. (2022) · Journal of Pregnancy · doi:10.1155/2022/8483777 · Cited by 32
Information disclosure to cancer patients in Mainland China: A meta‐analysis
Chen, C., CHENG, G., Chen, X., & Yu, L. (2022) · Psycho-Oncology · doi:10.1002/pon.6085 · Cited by 23
“The involved studies were assessed using the Mixed Methods Appraisal Tool (MMAT)—Version 2018, which was designed to appraise empirical studies for systematic literature reviews.(24) Five criteria were designed following two screening questions for assessing quantitative descriptive studies.”
MethodExtended Reality Use in Paediatric Intensive Care: A Scoping Review
Goldsworthy, A., Chawla, J., Baumann, O., Birt, J., & Gough, S. (2023) · Journal of Intensive Care Medicine · doi:10.1177/08850666231185721 · Cited by 26
Acute ischemic stroke outcomes in patients with COVID-19: a systematic review and meta-analysis
Ferrone, S. R., Sanmartin, M. X., O’Hara, J., Jimenez, J., Feizullayeva, C., Lodato, Z., Shahsavarani, S., Lacher, G., Demissie, S., Vialet, J., et al. (2023) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2023-020489 · Cited by 24
“23 Each study was evaluated by two independent reviewers for the screening and relevant questions, with discrepancies resolved by the senior independent arbitrator.”
BackgroundRobles-Palazón, F. J., Comfort, P., Ripley, N. J., Herrington, L. C., Bramah, C., & McMahon, J. J. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292487 · Cited by 76
“, brand, model, sampling rate, filters applied); on the other hand, the quality of the research conducted will be assessed by means of a modified version of the Mixed-Methods Appraisal Tool (MMAT) 2018 [44] as well as by means of a modified version of the Oxford Centre for Evidence-Based Medicine (OCEBM) 2009 model [4…”
MethodEngaging Parents in Technology-Assisted Interventions for Childhood Adversity: Systematic Review
Aldridge, G., Tomaselli, A., Nowell, C., Reupert, A., Jorm, A. F., & Yap, M. B. H. (2024) · Journal of Medical Internet Research · doi:10.2196/43994 · Cited by 31
“As results from the MMAT are best understood via ratings of each study design’s criterion rather than calculating an overall score per study [38], a brief summary of the key results is provided (Multimedia Appendix 5 provides a full summary of results and results from each included study).”
Public participation in healthcare students' education: An umbrella review
Nowell, L. S., Keogh, B., Laios, E., McKendrick-Calder, L. A., Molitor, W. L., & Wilbur, K. (2024) · Health Expectations · doi:10.1111/hex.13974 · Cited by 29
Collaborative Skills Training Using Digital Tools: A Systematic Literature Review
Cherbonnier, A., Hémon, B., Michinov, N., Jamet, É., & Michinov, E. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2348227 · Cited by 79
“We used the Mixed Methods Appraisal Tool (MMAT) to assess the quality of the studies (Hong et al., 2018).”
Raven-Gregg, T., Wood, F., & Shepherd, V. (2021) · Clinical Trials · doi:10.1177/1740774520988678 · Cited by 22
“Grey literature sources were also searched, and additional resources such as the ORRCA database (Online Resource for Recruitment Research in Clinical Trials).(18) Additional papers were also found by searching the reference lists of key relevant papers (the pearl-growing technique).”
BackgroundGutierrez, B. V., Kaloostian, D., & Redvers, N. (2023) · Current Developments in Nutrition · doi:10.1016/j.cdnut.2023.101973 · Cited by 21
Holm, A. L., Salemonsen, E., & Severinsson, E. (2021) · Nursing Open · doi:10.1002/nop2.789 · Cited by 21
“The third stage involved data evaluation, where the three authors evaluated the methodological quality of the included studies (Table 1, Table S1, Hong et al., 2018; Lotfi et al., 2019).”
MethodVerhoef, N. C., & Blomme, R. J. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1064889 · Cited by 27
“, 2020); the quality of the mixed methods studies according to MMAT procedure (Hong et al., 2018); the quality of the meta-analysis according to the 3-MIN procedure (Hussain et al.”
Method
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.