The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 1,801–1,900 of 4,578
Pre-hospital peripheral intravenous catheter insertion practice: An integrative review
Golling, E., van de Mortel, T., Barr, N., & Zimmerman, P.-A. P. (2022) · Australasian Emergency Care · doi:10.1016/j.auec.2022.08.006 · Cited by 8
Acceptability and Feasibility of Wearable Transdermal Alcohol Sensors: Systematic Review
Brobbin, E., Deluca, P., Hemrage, S., & Drummond, C. (2022) · JMIR Human Factors · doi:10.2196/40210 · Cited by 9
“For each included study, we determined the study design and then applied the appropriate screening criteria; this provided an overall quality score for the study [28].”
MethodInfluentialChallenges of interprofessional geriatric practice in home care settings: an integrative review
Song, Y., Jung, M. Y., Park, S., Hasnain, M., & Gruss, V. A. (2023) · Home Health Care Services Quarterly · doi:10.1080/01621424.2022.2164541 · Cited by 10
“For the two mixed-method studies, the five items addressing mixed-method study appraisal in the Mixed Methods Appraisal Tool (MMAT) were used (Hong et al., 2018).”
MethodOrjiakor, C. T., Eze, J. I., Ugwu, C. M., Ezenwa, M. O., Orjiakor, I. P., Onwujekwe, O. E., & Palamar, J. J. (2023) · Addiction Research & Theory · doi:10.1080/16066359.2023.2167982 · Cited by 9
Small, R., & Belluigi, D. Z. (2023) · Review Journal of Autism and Developmental Disorders · doi:10.1007/s40489-023-00366-6 · Cited by 14
“The risk of bias assessment, according to the MMAT protocols (Hong et al., 2018), placed 23 studies with a full high score of 5, one study with a score of 4.5, six studies with a score of 4, three studies with a score of 3.5, and one study with a score of 2.5.”
InfluentialExperiences of primary care among young adults with mental illness – A systematic literature review
Wickman, U. L., & Schmidt, M. (2023) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.13167 · Cited by 21
Sasseville, M., Ouellet, S., Rhéaume, C., Couture, V., Després, P., Paquette, J., Gentelet, K., Darmon, D., Bergeron, F., & Gagnon, M. (2023) · JMIR Research Protocols · doi:10.2196/46684 · Cited by 12
Self-citationBright, A., & Doody, O. (2023) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12943 · Cited by 12
Alshammari, A., & Brown, M. J. (2023) · Healthcare · doi:10.3390/healthcare11131857 · Cited by 9
“The MMAT contained five categories regarding the study designs, with each criterion rated as ‘yes’, ‘no’, or ‘cannot tell’ [17].”
MethodEffectiveness of nutrition interventions in Australian secondary schools: A systematic review
Comeau, A., Mertens, B., Bachwal, L., Utter, J., & van Herwerden, L. A. (2023) · Health Promotion Journal of Australia · doi:10.1002/hpja.787 · Cited by 7
Supportive care interventions for men with urological cancers: a scoping review
Saab, M. M., McCarthy, M. J., Murphy, M. F., Medved, K., O’Malley, M., Bambury, R. M., Gleeson, J. P., & Noonan, B. J. (2023) · Supportive Care in Cancer · doi:10.1007/s00520-023-07984-0 · Cited by 9
“The included studies were methodologically appraised using the Mixed Methods Appraisal Tool (MMAT) which assesses the quality of five study categories namely qualitative studies, RCTs, non-RCTs, quantitative descriptive studies and mixed methods studies [23].”
MethodUse of Assistive Technology for Persons with Psychosocial Disability: Systematic Review
Ebuenyi, I. D., Flocks-Monaghan, C., Rai, S. S., de Vries, R., Bhuyan, S. S., Pearlman, J. L., & Jones, N. (2023) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/49750 · Cited by 12
“The Mixed Methods Critical Appraisal Tool is a systematic appraisal tool where 2 independent reviewers score the quality of the included studies [22].”
MethodBarriers and facilitators to exoskeleton use in persons with spinal cord injury: a systematic review
Pinelli, E., Zinno, R., Barone, G., & Bragonzoni, L. (2023) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2023.2287153 · Cited by 8
“…methodological quality of the selected studies was independently assessed by two of the authors (eP and RZ) using the Mixed Methods Appraisal tool [26]. consensus was achieved through discussion with a 3rd author (lB) who acted as arbitrator if necessary. the Mixed Methods Appraisal tool (MMAt)…”
Self-medication with Novel Psychoactive Substances (NPS): a Systematic Review
Holborn, T., Page, R. L., II, Schifano, F., & Deluca, P. (2023) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-023-01195-8 · Cited by 10
“The included studies were subject to quality assessment using the MMAT and JBI checklists (Hong et al., 2018; The Joanna Briggs Institute, 2017).”
The unmet need for sexual health services among women with physical disabilities: a scoping review
Naghdi-Dorabati, P., Shahali, S., Montazeri, A., et al. (2024) · Journal of Public Health · doi:10.1007/s10389-024-02191-5 · Cited by 7
“To assess the quality of the articles, we used the Mixed Meth-ods Appraisal Tool (MMAT) to check the robustness of both qualitative and quantitative studies, and the PRISMA checklist for systematic reviews and meta-analyses (Hong et al. 2018; Fleming et al. 2014).”
Heslop, P., & Lauer, E. (2024) · British Journal of Learning Disabilities · doi:10.1111/bld.12576 · Cited by 12
Bitektine, E., Hintermayer, M., Chen, A., Ko, A., & Rodríguez, C. (2024) · Canadian Medical Education Journal · doi:10.36834/cmej.76338 · Cited by 11
Strydom, H., Delport, E., Muchiri, J. W., & White, Z. (2024) · Nutrients · doi:10.3390/nu16050584 · Cited by 10
“…(H.S. and J.M.) independently appraised the methodological quality of the articles according to five categories of research: qualitative research, quantitative randomised controlled trials, quantitative non-randomised studies, quantitative descriptive studies, and mixed methods studies [35].”
BackgroundMethodInfluentialMeasuring continuing medical education conference impact and attendee experience: a scoping review
Albrecht, L. M., Pratt, M., Ng, R., Olivier, J., Sampson, M., Fahey, N., Gibson, J., Lobos, A.-T., O’Hearn, K., Newhook, D., et al. (2024) · International Journal of Medical Education · doi:10.5116/ijme.65cc.8c88 · Cited by 10
Thursby, S., Dismore, L. L., & Swainston, K. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05306-4 · Cited by 8
“The quality of included studies was assessed using the Mixed Methods Assessment Tool (MMAT) for systematic multi-method reviews [27].”
MethodGrattidge, L., Hoang, D. H., Lees, D., Visentin, D. C., Mond, J., & Auckland, S. (2024) · Mental Health & Prevention · doi:10.1016/j.mhp.2024.200335 · Cited by 10
Empathy in family medicine postgraduate education: A mixed studies systematic review
Ortiz-Paredes, D., Henet, P. A., Desseilles, M., & Rodríguez, C. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2328324 · Cited by 9
“DOP, PAH, and CR used the 2018 version of the Mixed Methods Appraisal Tool (MMAT) (Pace et al. 2012; Hong et al. 2018a, 2018b, 2019; Pluye and Hong 2019)”
Lakshman, R., Tomlinson, E., & Bucknall, T. K. (2024) · Pain Management Nursing · doi:10.1016/j.pmn.2024.03.008 · Cited by 8
Naya, K., Sakuramoto, H., Aikawa, G., Ouchi, A., Yoshihara, S., Ota, Y., Okamoto, S., Fukushima, A., & Hirashima, H. (2024) · Cureus · doi:10.7759/cureus.58344 · Cited by 7
“Further, two of the nine reviewers independently appraised the studies during the data extraction phase using the Mixed Methods Assessment Tool (MMAT) [12].”
MethodBingham, S. L., Cassells, C. V., & Semple, C. J. (2024) · Sexuality Research and Social Policy · doi:10.1007/s13178-024-00974-9 · Cited by 8
“The MMAT includes two screening questions and five methodological questions specific to study design types.”
InfluentialSalter, J., & Blainey, S. H. (2024) · BMC Psychology · doi:10.1186/s40359-024-01770-1 · Cited by 15
Mudiyanselage, C. A. K. R. A., Ewens, B. A., Smyth, A. M., Dickson, J. M., & Ang, S. G. M. (2024) · Mindfulness · doi:10.1007/s12671-024-02365-y · Cited by 9
“Critical appraisal of included studies was conducted using the Mixed Methods Appraisal Tool (MMAT), which was designed to appraise quantitative, qualitative, and mixed-methods studies (Hong et al., 2018).”
Mullen, C., Howard, E., & Cronin, A. G. (2024) · Educational Studies in Mathematics · doi:10.1007/s10649-024-10332-6 · Cited by 12
“It was planned to evaluate the quality of the methodology and therefore the risk of bias in the studies included in the review using the Mixed-Methods Appraisal Tool (MMAT) version 2018 (Hong et al., 2018).”
InfluentialCodesign Use in Palliative Care Intervention Development: A Systematic Review
Giannitrapani, K. F., Lin, K., Hafi, L. A., Maheta, B. J., & Isenberg, S. R. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.06.007 · Cited by 7
Ekici, H. S., Collins, J., Kafadar, A. H., Yildirim, M. C., Phillips, B. E., & Gordon, A. L. (2024) · European Geriatric Medicine · doi:10.1007/s41999-024-01028-4 · Cited by 8
“The MMAT provides a quality score from 0 to 5, with a score of 4 or 5 considered high quality, and a score of < 3 low quality [28].”
Pharmacy practice and policy research in Türkiye: a systematic review of literature
Gülpınar, G., Pehlivanlı, A., & Babar, Z. (2024) · Journal of Pharmaceutical Policy and Practice · doi:10.1080/20523211.2024.2385939 · Cited by 7
“The MMAT incorporates two primary screening criteria to evaluate the formulation of a distinct research question/objective and the suitability of the gathered data for addressing the research query.”
MethodInfluentialPolden, M., Faulkner, T. C., Holland, C., Hanna, K. L., Ward, K., Ahmed, F., Brown, H., Barrow, H., Main, J., Mann, S., et al. (2024) · Dementia · doi:10.1177/14713012241273837 · Cited by 15
“We assessed the risk of bias using the Mixed Methods Appraisal Tool (MMAT) which has criteria for 5 study designs: qualitative, quantitative randomized controlled, quantitative nonrandomized, quantitative descriptive, and mixed methods (Hong et al., 2018).”
Peterson, K. T., Wilson, O. W. A., Herrick, S. S. C., Frederick, G. M., Fedewa, M. V., Sullivan, K. W., & Bopp, M. (2024) · Journal of Physical Activity and Health · doi:10.1123/jpah.2024-0336 · Cited by 10
Gilchrist, A. I., Fernando, G. V. M. C., Holland, P., & Ahmed, F. (2024) · PLoS ONE · doi:10.1371/journal.pone.0314620 · Cited by 9
“The quality of quantitative and qualitative studies was assessed using the Mixed Methods Appraisal Tool (MMAT) (2018) [38].”
InfluentialHolmen, H., Flølo, T. N., Tørris, C., Torbjørnsen, A., Almendingen, K., & Riiser, K. (2025) · Frontiers in Pediatrics · doi:10.3389/fped.2024.1507379 · Cited by 11
O’Connell, M. A., Meedya, S., Baqali, J. A., Alraeesi, K., & Leahy‐Warren, P. (2025) · International Breastfeeding Journal · doi:10.1186/s13006-025-00728-2 · Cited by 7
“[37].”
O'Mahony, B., Kerins, C., Murrin, C. M., & Kelly, C. N. M. (2020) · HRB Open Research · doi:10.12688/hrbopenres.13041.3 · Cited by 10
Attitudes and perceptions of health professionals towards sleep health: a systematic review
Caudwell, L., Himani, H., Khaw, A., Taylor, R., White, J., Rhodes, S., & Skinner, M. A. (2020) · Physical Therapy Reviews · doi:10.1080/10833196.2020.1832713 · Cited by 10
Varela, S. M., Hays, C. A., Knight, S. M., & Hays, R. (2021) · Australian Journal of Rural Health · doi:10.1111/ajr.12740 · Cited by 7
de Graaf, L., Janssen, M. M., Roelofs, T. S. M., & Luijkx, K. G. (2021) · Ageing and Society · doi:10.1017/s0144686x21001215 · Cited by 19
“The quality of the included papers was assessed by three researchers independently (LG, MJ and TR) using the Mixed Methods Assessment Tool (MMAT; Hong et al .”
MethodRoostaei, M., Dalvand, H., Rassafiani, M., Kelly, G., & Razi, B. (2022) · Canadian Journal of Occupational Therapy · doi:10.1177/00084174211066651 · Cited by 9
“According to MMAT, three studies were determined as “High” (Cameron et al., 2017; Ghorbani et al., 2017; Sousa et al., 2020) and four studies were determined as having “moderate” methodological quality (Gimeno et al., 2019,2021; Jackman et al., 2018a, 2018b).”
MethodInfluentialCoombs, C., Cohen, T., Duddy, C., Mahtani, K. R., Roberts, N. W., Saini, A., Foster, A. S., & Park, S. E. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-052651 · Cited by 9
Protocol of a multiphase study on telemedicine for older adults in primary care
Khanassov, V., Ilali, M., & Vedel, I. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057061 · Cited by 8
Self-citationWilliams, J. (2022) · Journal of Social Work Practice · doi:10.1080/02650533.2022.2089639 · Cited by 13
McCahill, R. J., Keogh, S. J., & Hughes, J. A. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16520 · Cited by 8
Patient‐mediated interventions in hospital: A systematic review
Tobiano, G., Roberts, S., Muir, R., Jerofke‐Owen, T. A., Ting, C., Thorning, S., Heyland, D. K., & Marshall, A. P. (2022) · Journal of Advanced Nursing · doi:10.1111/jan.15500 · Cited by 7
“This tool enables critical appraisal of diverse study designs and has demonstrated validity and reliability (Hong, Fàbregues, et al., 2018).”
MethodJahan, I., Dalal, K., Khan, M. A. S., et al. (2022) · JMIR Research Protocols · doi:10.2196/42239 · Cited by 7
“…Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines, use reliable tools to assess study quality [33,34], and propose a method to query the literature about the occupational exposures and health hazards among South Asian traffic police.”
Indicators for Modernising the LIS Internship: Evidences from a systematic literature review
Azizi, N. A. N., & Kaur, K. (2023) · Journal of Librarianship and Information Science · doi:10.1177/09610006231154536 · Cited by 9
“To gauge this purpose, an evaluation tool called the Mixed-Method Appraisal Tool (MMAT) by Hong et al. (2018) was used.”
Almashmoum, M., Cunningham, J. A., Alkhaldi, O., & Anisworth, J. (2023) · JMIR Human Factors · doi:10.2196/44327 · Cited by 8
“Following the final selection of studies, the risk of bias was assessed using the Mixed Methods Appraisal Tool (MMAT) [38].”
MethodInfluentialHong, J. Q. Y., Chua, W. L., Smith, D., Huang, C., Goh, Q. L. P., & Liaw, S. Y. (2023) · Journal of Clinical Nursing · doi:10.1111/jocn.16743 · Cited by 8
Walsh‐Bailey, C., Gilbert, A. S., Shato, T., Sandler, B., Baumann, A. A., Bradley, C., McLoughlin, G. M., McGuire, F. H., Fort, M. P., & Tabak, R. G. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02240-2 · Cited by 11
“We will apply the Mixed Methods Appraisal Tool (MMAT) to evaluate study quality during extraction [77].”
MethodSafety climate in the surgical center during the Covid-19 pandemic: mixed-method study
Azevedo, A. R. R., Fassarella, C. S., Lourenção, D. C. D. A., Camerini, F. G., Henrique, D. D. M., & da Silva, R. F. A. (2023) · BMC Nursing · doi:10.1186/s12912-023-01358-x · Cited by 11
“research, Consolidated Criteria for Reporting Qualitative Research (COREQ): a 32-item checklist for interviews and focus groups [17]; and, so as to provide even greater methodological rigour in the mixed-method study, the Mixed Methods Appraisal Tool (MMAT) [18, 19].”
MethodLong-Tailed Macaque (Macaca fascicularis) Contraception Methods: A Systematic Review
Mikail, M., Azizan, T. R. P. T., Noor, M. H. M., Hassim, H. A., Che-Amat, A., & Latip, M. Q. A. (2023) · Biology · doi:10.3390/biology12060848 · Cited by 11
“Furthermore, the methodological quality of each article was evaluated using the Mixed Method Appraisal Tool (MMAT) and sensitivity analysis, as described in [24].”
MethodJackson, N., Turner, M. R., & Paterson, C. C. (2023) · International Journal of Older People Nursing · doi:10.1111/opn.12557 · Cited by 7
Nabhan, A. F., Kabra, R., Ashraf, A., Elghamry, F., Kiarie, J. N., Atwa, E., bahnasy, S., Elgendi, M., Elshabrawy, A., Eltayeb, S., et al. (2023) · BMC Women s Health · doi:10.1186/s12905-023-02735-z · Cited by 7
“We accepted that there is no ‘gold standard’ approach for assessing the methodo - logical quality of primary qualitative studies, but believe that MMAT fits the context of this synthesis [19, 20].”
BackgroundMethodChan, D. N. S., Li, C., Law, B. M. H., Xu, B., & Kwok, C. (2023) · Journal of Immigrant and Minority Health · doi:10.1007/s10903-023-01564-4 · Cited by 10
“The methodological quality of the studies was assessed using the Mixed Methods Appraisal Tool [17].”
van Duinkerken, A., Bosmans, M. W. G., Baliatsas, C., Tak, N., Meerdink, A., Jansen, N., Mahon, M. D. V.-M., Marra, E., & Dückers, M. (2023) · Methods and Protocols · doi:10.3390/mps6060117 · Cited by 6
“In the first year, a combination of several validated instruments (the AXIS Quality Assessment tool [27], the Mixed Methods Ap-praisal Tool [28], and the Critical Appraisal Programme [29]) is used for quality assessment.”
MethodTelehealth technologies in care homes: a gap for dentistry?
Bhamra, I. B., Gallagher, J. E., & Patel, R. (2023) · Journal of Public Health · doi:10.1093/pubmed/fdad258 · Cited by 9
Rodriguez, A., Shambhunath, S., Wijesiri, T. I. D., Dalcin, C. B., & Goldrick, N. M. (2024) · Frontiers in Oral Health · doi:10.3389/froh.2024.1355349 · Cited by 9
“In addition, Mullins et al. and Burrows et al. reinforced how participation in those studies made participants feel their voices were heard and valued (24, 25).”
MethodInfluentialDecision-making aids for upper limb interventions in neurological rehabilitation: a scoping review
Tan, A., Copley, J., & Fleming, J. (2021) · Disability and Rehabilitation · doi:10.1080/09638288.2021.1924881 · Cited by 7
Wicklum, S., Willis, E., Amson, A., McGuire, K. A., Crowshoe, L. L., McBrien, K., & Henderson, R. (2021) · SAGE Open · doi:10.1177/21582440211016845 · Cited by 7
“The Mixed Methods Appraisal Tool (MMAT) was used to assess study quality (Hong et al., 2018) (Supplementary Table 1).”
MethodNurses’ views and practices on parental mental illness: An integrative review
Vives-Espelta, J., Burjalés-Martí, M. D., & Ferré‐Grau, C. (2021) · International Journal of Mental Health Nursing · doi:10.1111/inm.12937 · Cited by 7
Global trends in occupational disease reporting: a systematic reviewPreprint
Khoe, L. C., Saldi, S. R. F., Isbayuputra, M., Mansyur, M., Wiseman, V. L., & Asante, A. D. (2024) · medRxiv · doi:10.1101/2024.09.19.24314032 · Cited by 7
Hilton, S. P., An, N. H., O’Brien, L. A., Snyder, J. S., Rogers, H., Cumming, O., Mills, J. E., Gordon, B. A., Freeman, M. C., Caruso, B. A., et al. (2025) · BMJ Global Health · doi:10.1136/bmjgh-2025-018925 · Cited by 13
Vhora, A., Davies, R. L., & Rice, K. (2023) · Psychology Learning & Teaching · doi:10.1177/14757257231196707 · Cited by 6
“For quantitative studies, the MMAT evaluates sampling, construct measurement, nonresponse bias, and statistical analyses, and for qualitative studies it evaluates data collection, interpretation of fi ndings, and overall coherence.”
MethodInfluentialLombardo, C., & Islam, M. S. (2023) · Physical Therapy Reviews · doi:10.1080/10833196.2023.2271301 · Cited by 8
“Each criterion is rated as ‘Yes’ ‘No’ or ‘Can’t tell’, and provides an overall evaluation of the methodological quality and risk of bias of the study [22].”
BackgroundMethodInfluentialHaltaufderheide, J., Lucht, A., Strünck, C., & Vollmann, J. (2023) · BMC Medical Ethics · doi:10.1186/s12910-023-00984-z · Cited by 7
“Although not planned in advance and with the protocol, we decided to appraise methodological quality using the MMAT tool for mixed-methods reviews [35].”
MethodVölker, M., & Sujaritpong, S. (2024) · Air Quality Atmosphere & Health · doi:10.1007/s11869-024-01531-4 · Cited by 9
“The quality of the articles was assessed by the two authors of this paper using different sets of criteria for quantitative, qualitative, mixed-methods and WTP studies that were adapted from Hong et al. (2018), Kmet et al. (2004) and Söderqvist and Soutukorva (2006).”
Santonja-Ayuso, L., Portillo, M. C., & Corchón, S. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21040485 · Cited by 12
“The MMAT guideline was used in studies with mixed methodology [30].”
MethodSibling experiences in the neonatal intensive care unit: An integrative review
Trajkovski, S., & Tehrani, M. B. (2024) · Journal of Neonatal Nursing · doi:10.1016/j.jnn.2024.07.021 · Cited by 6
Care of Older Adults with Mental Illness in Long-Term Care Residential Facility: A Scoping Review
Čížková, J., Dostálová, V., Bártová, A., Holmerová, I., & Jelínková, P. V. (2024) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2024.105218 · Cited by 9
Hybrid mHealth care: Patient perspectives of blended treatments for psychosis. A systematic review
Greenway, F. T., Weal, M. J., & Palmer-Cooper, E. C. (2024) · Schizophrenia Research · doi:10.1016/j.schres.2024.08.017 · Cited by 11
“The Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018), sections 1 and 5 were chosen to assess the quality assessment of the included studies.”
MethodThe reliability and validity of triage tools in geriatric emergency departments: A scoping review
Zhang, W., Zhang, M., Yang, P., Zhou, W., Zheng, J., & Zhang, Y. (2024) · International Emergency Nursing · doi:10.1016/j.ienj.2024.101509 · Cited by 10
Outpatient parenteral antimicrobial therapy with carbapenems: A systematic review
Wolie, Z. T., Roberts, J. A., Wale, Y. M., et al. (2024) · Journal of Infection · doi:10.1016/j.jinf.2024.106299 · Cited by 6
Kearns, Á., Gallagher, A., Hayes, A., O'Gorman, A., O’Muiri, S., & O’Rourke, L. (2024) · The International Journal of Evidence & Proof · doi:10.1177/13657127241287322 · Cited by 8
Nguyen, H., Rahman, A., Ubell, A., Goodarzi, Z., Maxwell, C. J., Allana, S., Tate, K., Symonds‐Brown, H., Weeks, L. E., Caspar, S., et al. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02683-1 · Cited by 10
“…of participants [95, 96], (2) The Newcastle–Ottawa Scale (NOS) for cohort studies or case–control studies [97], (3) Appraisal tool for Cross-Sectional Studies (AXIS) [98], (4) Critical Appraisal Skills Program (CASP) Qualitative Studies Checklist [99], (5) Mixed Methods Appraisal Tool (MMAT) [100].”
MethodMachinga-Asaolu, R. O. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241291075 · Cited by 9
“Eligible articles were further screened for quality assurance using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
Kellett, P., Franklin, B. D., Pearce, S., & Benn, J. J. (2024) · BMJ Open Quality · doi:10.1136/bmjoq-2023-002711 · Cited by 12
“Data were extracted and recorded using a bespoke tool developed through author piloting and review, displaying aspects of core details, study background and results.”
Jans, C., Levett-Jones, T. L., & Lucas, C. (2024) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2024.101626 · Cited by 9
Georgopoulou, S., Droney, J., Jaganathan, P. P., Howell, P., Doherty, A. M., Young, K., & Cruickshank, S. (2024) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2024.102862 · Cited by 7
Ang, S. Y., Saat, G. A. M., & Shittu, O. T. (2025) · Mental Health & Prevention · doi:10.1016/j.mhp.2025.200405 · Cited by 14
The role of Experiential Avoidance and related factors in eating disorders: a systematic review
Gulacan, D., Gunay-Oge, R., & İnözü, M. (2025) · Current Psychology · doi:10.1007/s12144-025-07425-7 · Cited by 10
“Bias assessment of included studies was performed using the MMAT and no studies were excluded on the basis of quality assessment.”
InfluentialPrue, G., Czamanski‐Cohen, J., Kassianos, A. P., Pilleron, S., Ladas, A. I., Can, A. A., Roldán‐Jiménez, C., Dégi, C. L., İŞCAN, G., Arslan, S., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-085456 · Cited by 8
“59 This tool is most suitable as it can be used for five different types of designs: qualitative research, randomised controlled trials, non-randomised studies, quantitative descriptive studies and mixed-methods studies.”
Technologies and Applications of Robots in Dementia Care: A Systematic Review
Wu, D., Pu, L., Jo, J., & Moyle, W. (2025) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-025-02232-5 · Cited by 12
“The use of a quality assessment tool [32] serves as a guideline for engineers to work with health researchers to develop rigorous studies for clinical practice.”
Narrative medicine in pediatric medical education and patient care: A scoping review
Tseng, T., Kuo, P.-Y., Lin, M.-J., & Chu, S. (2025) · Tzu Chi Medical Journal · doi:10.4103/tcmj.tcmj_181_24 · Cited by 9
“Studies achieving 4 or 5 “Yes” ratings are generally considered high quality, whereas those with 2 or fewer were deemed suboptimal [26].”
Cooke, S., Nelson, D., Argin, A. A., Laparidou, D., Young, R. H., Waller, J., Kane, R., McInnerney, D., Quaife, S. L., Peake, M. D., et al. (2025) · Lung Cancer · doi:10.1016/j.lungcan.2025.108484 · Cited by 8
Shahdan, S., & Sidek, S. (2025) · Appetite · doi:10.1016/j.appet.2025.107979 · Cited by 9
Gallagher, A., Mainar, P. D., Cronin, C., Muñoz, C., Calleja, J. R., Loughnane, C., & Trujillo, J. (2025) · Pediatric Allergy and Immunology · doi:10.1111/pai.70075 · Cited by 12
Barriers and facilitators to pediatric tuberculosis management in India: a systematic review
Willis, M. D., de Wetering, J. V., Brown, H., et al. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-025-10863-0 · Cited by 6
Alegre, E., Penman, A. M., Unicomb, R., & Scarinci, N. (2025) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.70040 · Cited by 8
Brown, A., Cant, A., Wolfgang, R., Ramsden, R., Heaney, S. E., & Brown, L. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12717-6 · Cited by 8
“While methodological criteria are more difficult to assess, it may be considered more rigorous than simply reporting quality [24].”
Weakly supervised text classification on free-text comments in patient-reported outcome measures
Linton, A.-G., Dimitrova, V. G., Downing, A., Wagland, R., & Glaser, A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1345360 · Cited by 9
The impact of multimorbidity on suicidal behaviour: A systematic review and meta-analysis
Opoku, R., Mensah, A. K., & Nath, M. (2025) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2025.04.008 · Cited by 12
Crosland, S., Tzani, C., & Ιωάννου, Μ. (2025) · Deviant Behavior · doi:10.1080/01639625.2025.2501618 · Cited by 9
Systematic Review of Psychodermatologic Assessment Tools: Diagnostic Accuracy and Clinical Utility
Clark, C., Abdi, P., Li, K. D., Turk, T., & Dytoc, M. T. (2025) · Journal of Cutaneous Medicine and Surgery · doi:10.1177/12034754241311267 · Cited by 12
Spirituality, Religion, and Sports Injury: A Systematic Review
Liu, S., Noh, Y., Mahfooz, M., & Hassan, M. F. H. (2025) · Journal of Religion and Health · doi:10.1007/s10943-025-02323-5 · Cited by 9
“A methodological quality appraisal using the mixed methods evaluation tool (MMAT) was used to appraise the quality of empirical studies (Hong et al., 2018).”
Munnangi, M., Shreedhar, P., Lippus, H., Gilyan, R., Evans, D. P., & Maxwell, L. (2025) · Frontiers in Reproductive Health · doi:10.3389/frph.2025.1561707 · Cited by 8
Fatai, A., Talerd, W., Siriprathum, S., et al. (2025) · Journal of Emergency Nursing · doi:10.1016/j.jen.2025.04.007 · Cited by 6
Enticott, A., & Dew, A. (2025) · Journal of Intellectual & Developmental Disability · doi:10.3109/13668250.2025.2499674 · Cited by 7
“The Effective Public Health Practice Project (EPHPP) (https://www.ephpp. ca/quality-assessment-tool-for-quantitative-studies/) tool was selected to appraise the quantitative studies, and the Mixed Methods Appraisal Tool version 18 (MMAT) (Hong et al., 2018) was used to appraise the qualitative and mixed methods studie…”
Vázquez-Lara, M. D., Rúger-Navarrete, A., Mohamed-Abdel-Lah, S., Gómez‐Urquiza, J. L., Fernández-Carrasco, F. J., Díaz, L. R., Caparrós-González, R. A., Palomo-Gómez, R., Riesco-González, F. J., & Vázquez-Lara, J. M. (2025) · Healthcare · doi:10.3390/healthcare13121378 · Cited by 19
“In the critical reading process, five questions from the Mixed Methods Appraisal Tool (MMAT) checklist (2018 version) were considered for clinical trials or quasi-experimental studies, depending on the study [28], once both screening questions were answered affirmatively.”
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.