The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 101–200 of 4,578
Usher, K. J., Jackson, D. W., Durkin, J., Gyamfi, N., & Bhullar, N. (2020) · International Journal of Mental Health Nursing · doi:10.1111/inm.12790 · Cited by 113
Hudon, C., Chouinard, M.-C., Pluye, P., El Sherif, R., Bush, P. L., Rihoux, B., Poitras, M.-E., Lambert, M., Zomahoun, H. T. V., & Légaré, F. (2019) · The Annals of Family Medicine · doi:10.1370/afm.2419 · Cited by 90
Self-citationWesarg, C., Ebbes, R., Hensums, M., Wagemaker, E., Zaharieva, M. S., Staaks, J. P. C., van den Akker, A. L., Visser, I., Hoeve, M., Brummelman, E., et al. (2023) · Developmental Review · doi:10.1016/j.dr.2023.101090 · Cited by 144
Zhuang, S., Tan, D. W., Reddrop, S., Dean, L., Maybery, M., & Magiati, I. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2023.102335 · Cited by 90
Huda, E., Hawker, P. J., Cibralic, S., John, J. R., Hussain, A., Diaz, A. M., & Eapen, V. (2024) · BMC Pediatrics · doi:10.1186/s12887-024-05067-5 · Cited by 100
“Two reviewers (EH and PH) conducted separate evaluations of the study quality using either the Quality Assessment of Diagnostic Accuracy Studies Tool (QUADAS-2) [29] or the Mixed Methods Appraisal Tool (MMAT) [30].”
MethodHow to … do mixed‐methods research
Kajamaa, A., Mattick, K., & de la Croix, A. (2020) · The Clinical Teacher · doi:10.1111/tct.13145 · Cited by 90
Tully, L., Case, L., Arthurs, N., Sorensen, J., Marcin, J. P., & O'Malley, G. (2021) · Frontiers in Pediatrics · doi:10.3389/fped.2021.630365 · Cited by 90
“Abbreviations: HCP, Healthcare Professional; ICT, Information and Communication Technology; MMAT, Mixed-Methods Appraisal Tool; CFIR, Consolidated Framework for Implementation Research.”
MethodInfluentialTseole, N. P., Mindu, T., Kalinda, C., & Chimbari, M. J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0271726 · Cited by 107
“We assessed all selected articles for quality using a mixed methods appraisal tool (MMAT) [15].”
MethodYopa, D. S., Massom, D. M., Kiki, G. M., Sophie, R. W., Sylvie, F., Thiam, O., Zinaba, L., & Ngangue, P. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1252428 · Cited by 95
Making change last? Exploring the value of sustainability approaches in healthcare: a scoping review
Lennox, L., Linwood-Amor, A., Maher, L., & Reed, J. E. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-00601-0 · Cited by 110
Massively Multiplayer Online Games and Well-Being: A Systematic Literature Review
Raith, L., Bignill, J., Stavropoulos, V., Millear, P., Allen, A., Stallman, H. M., Mason, J., De Regt, T., Wood, A., & Kannis-Dymand, L. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.698799 · Cited by 89
“The Mixed Methods Appraisal Tool (Hong et al., 2018) was used by JB and LR to conduct their independent appraisals of each study.”
MethodAfaya, A., Konlan, K. D., & Kim, H. O. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07187-5 · Cited by 145
AI in the Health Sector: Systematic Review of Key Skills for Future Health Professionals
Gazquez-Garcia, J., Sánchez-Bocanegra, C. L., & Sevillano, J. L. (2025) · JMIR Medical Education · doi:10.2196/58161 · Cited by 88
Austin, J., Drossaert, C. H. C., Schroevers, M. J., Sanderman, R., Kirby, J. N., & Bohlmeijer, E. T. (2021) · Psychology & Health · doi:10.1080/08870446.2019.1699090 · Cited by 88
“The MMAT is designed to appraise the quality of studies in a review process based on criteria appropriate to the type of design.”
BackgroundMethodInfluentialXu, Y., Lau, Y., Cheng, L. J., & Lau, S. T. (2021) · Nurse Education Today · doi:10.1016/j.nedt.2021.105139 · Cited by 106
Guemghar, I., Padilha, P. P. D. O., Abdel‐Baki, A., Jutras‐Aswad, D., Paquette, J., & Pomey, M. (2022) · JMIR Mental Health · doi:10.2196/36094 · Cited by 125
Schmid, M., Brianza, E., Mok, S. Y., & Petko, D. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105024 · Cited by 87
“Regarding the assessment of the methodological quality of reported primary studies with qualitative designs within the reviews and meta-analyses, different tools that might be eligible to assess the methodological quality of studies were screened (e.g., Hong et al., 2018; Popay et al., 1998).”
Physical workplace adjustments to support neurodivergent workers: A systematic review
Weber, C., Krieger, B., Häne, E., Yarker, J., & McDowall, A. (2024) · Applied Psychology · doi:10.1111/apps.12431 · Cited by 86
Dariotis, J. K., Chen, F. R., Park, Y. R., Nowak, M. K., French, K. M., & Codamon, A. M. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20136197 · Cited by 86
“Given these high MMAT criteria ratings, it was concluded that the risk of bias was low, and the study quality was high for the qualitative/mixed studies included in this review.”
MethodInfluentialAutistic identity: A systematic review of quantitative research
Davies, J., Cooper, K., Killick, E., Sam, E., Healy, M., Thompson, G. A., Mandy, W., Redmayne, B. J. M., & Crane, L. (2024) · Autism Research · doi:10.1002/aur.3105 · Cited by 123
“Dis-agreements were discussed, referring back to the MMAT guidance where necessary, to reach consensus.”
MethodInfluentialBerger, S., Grzonka, P., Frei, A. I., Hunziker, S., Baumann, S. M., Amacher, S. A., Gebhard, C. E., & Sutter, R. (2024) · Critical Care · doi:10.1186/s13054-024-04844-z · Cited by 116
“…risk of bias and methodological quality of the included studies as follows: Two reviewers (SB and RS) used the Mixed Methods Appraisal Tool (MMAT) [19] for cross-sectional studies and the Newcastle Ottawa Quality Assessment Scale for observational studies [20] to assess the quality of the…”
MethodRiley, B. J., Oster, C., Rahamathulla, M., & Lawn, S. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18030984 · Cited by 130
“Quality appraisals were conducted using the Mixed Methods Appraisal Tool (MMAT) for qualitative and quantitative randomised controlled trials, quantitative non-randomised trials, quantitative descriptive studies, and mixed methods studies [17].”
MethodA systematic review of multimodal prehabilitation in breast cancer
Toohey, K. L., Hunter, M., McKinnon, K., Casey, T., Turner, M. R., Taylor, S., & Paterson, C. C. (2022) · Breast Cancer Research and Treatment · doi:10.1007/s10549-022-06759-1 · Cited by 106
“Methodological quality assessment of the included studies was completed using the Mixed Methods Appraisal Tool (MMAT 2018) [36].”
MethodLouwen, C., Reidlinger, D. P., & Milne, N. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04003-y · Cited by 136
Loneliness in the workplace: a mixed-method systematic review and meta-analysis
Bryan, B. T., Andrews, G., Thompson, K. N., Qualter, P., Matthews, T., & Arseneault, L. (2023) · Occupational Medicine · doi:10.1093/occmed/kqad138 · Cited by 124
“The methodological quality of each article was assessed by two researchers (B.T.B. and G.A. or K.N.T.) independently using the Mixed-Methods Appraisal Tool [14].”
MethodThe physical, psychological and social impact of long bone fractures on adults: A review
Singaram, S., & Naidoo, M. (2019) · African Journal of Primary Health Care & Family Medicine · doi:10.4102/phcfm.v11i1.1908 · Cited by 97
Butler-Henderson, K. A., Crawford, J., Rudolph, J., Lalani, K. H., & Sabu, K. M. (2020) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2020.3.2.11 · Cited by 93
eHealth in Geriatric Rehabilitation: Systematic Review of Effectiveness, Feasibility, and Usability
Kraaijkamp, J. J. M., van Isselt, E. F. V. D., Persoon, A., Versluis, A., Chavannes, N. H., & Achterberg, W. P. (2021) · Journal of Medical Internet Research · doi:10.2196/24015 · Cited by 103
“The quality of included studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [36], which allowed quality assessment across different study designs.”
The Mental Health Impacts of COVID-19 on PK–12 Students: A Systematic Review of Emerging Literature
Naff, D., Williams, S. E., Furman, J., & Yeung, M. (2022) · AERA Open · doi:10.1177/23328584221084722 · Cited by 106
“This process also included a quality appraisal using the Mixed-Methods Appraisal Tool (Hong et al., 2018).”
MethodAbusubhiah, M., Walshe, N., Creedon, R., Noonan, B. J., & Hegarty, J. M. (2023) · Nursing Open · doi:10.1002/nop2.1931 · Cited by 112
Anderson, E. J., Krause, K. C., Krause, C. M., Welter, A., McClelland, D. J., Garcia, D. O., Ernst, K. C., Lopez, E. C., & Koss, M. P. (2019) · Trauma Violence & Abuse · doi:10.1177/1524838019888889 · Cited by 101
“Two authors independently performed a study-level risk of bias appraisal using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) to address qualitative, quantitative, and mixed-methods outcomes and to estimate the overall scientific strength of evidence available on the study topic (Online Appendix).”
MethodGreen spaces, dementia and a meaningful life in the community: A mixed studies review
Mmako, N. J., Courtney‐Pratt, H., & Marsh, P. (2020) · Health & Place · doi:10.1016/j.healthplace.2020.102344 · Cited by 104
Tan, A. J. Q., Rusli, K. D. B., McKenna, L., Tan, L. L. C., & Liaw, S. Y. (2021) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x211049206 · Cited by 103
“28 (Supplemental material 3).”
Factors affecting the UK junior doctor workforce retention crisis: an integrative review
Lock, F. K., & Carrieri, D. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059397 · Cited by 102
Strategies and tactics to reduce the impact of healthcare on climate change: systematic review
Braithwaite, J., Smith, C. L., Leask, E., Wijekulasuriya, S., Brooke-Cowden, K., Fisher, G., Patel, R., Pagano, L., Rahimi‐Ardabili, H., Spanos, S., et al. (2024) · BMJ · doi:10.1136/bmj-2024-081284 · Cited by 120
How Much of Root Cause Analysis Translates into Improved Patient Safety: A Systematic Review
Martín-Delgado, J., Martínez-García, A., Andrés, J. M. A., et al. (2020) · Medical Principles and Practice · doi:10.1159/000508677 · Cited by 81
Muthuri, R. N. D. K., Senkubuge, F., & Hongoro, C. (2020) · Healthcare · doi:10.3390/healthcare8020164 · Cited by 120
“For mixed methods studies, the Mixed Methods Appraisal Tool (MMAT) [26], was used for assessment of the same.”
MethodGasteiger, N., van der Veer, S. N., Wilson, P. M., & Dowding, D. (2021) · JMIR Serious Games · doi:10.2196/31644 · Cited by 127
“The MMAT consists of 2 screening questions and 5 study design–specific criteria that could be scored 1 (yes) or 0 (no) [51].”
BackgroundMethodChen, J., Zhang, S. X., Yin, A., & Yáñez, J. A. (2022) · Journal of Global Health · doi:10.7189/jogh.12.05011 · Cited by 100
Brandt, M., Johannsen, L. M., Inhestern, L., & Bergelt, C. (2022) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-022-02407-5 · Cited by 108
Ade-Ojo, G. O., Markowski, M., Essex, R., Stiell, M., & Jameson, J. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12653 · Cited by 104
Intelligent Physical Robots in Health Care: Systematic Literature Review
Huang, R., Li, H., Suomi, R., Li, C., & Peltoniemi, T. (2022) · Journal of Medical Internet Research · doi:10.2196/39786 · Cited by 98
“To evaluate the methodological quality of the studies, the first and second authors appraised the quality of the selected articles independently according to the Mixed Methods Appraisal Tool (MMAT; version 2018) [40], which can be applied to evaluate empirical studies using 5 different research methods: qualitative me…”
MethodUnderreporting of workers’ injuries or illnesses and contributing factors: a systematic review
Kyung, M., Lee, S., Dancu, C., & Hong, O. (2023) · BMC Public Health · doi:10.1186/s12889-023-15487-0 · Cited by 83
“Study risk of bias assessment The risk of bias of included studies was appraised by the two reviewers who are enrolled in a PhD program, using the Mixed Methods Appraisal Tool (MMAT), a valid and reliable measure of systematic review of mixed method studies [11, 12].”
BackgroundMethodInfluentialSelf-Reported Symptoms of Depression and Anxiety After ACL Injury: A Systematic Review
Piussi, R., Berghdal, T., Sundemo, D. S., Grassi, A., Zaffagnini, S., Sansone, M., Samuelsson, K., & Senorski, E. H. (2022) · Orthopaedic Journal of Sports Medicine · doi:10.1177/23259671211066493 · Cited by 94
“A critical appraisal of the included qualitative studies was performed using the Mixed-Methods Appraisal Tool (MMAT).(26) The MMAT includes 2 screening questions and assesses studies according to category based on the study methods.”
MethodTruong, M., Yeganeh, L., Cook, O., Crawford, K., Wong, P., & Allen, J. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac015 · Cited by 90
Boyle, L. D., Husebo, B. S., & Vislapuu, M. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08968-2 · Cited by 108
“To reduce the risk of bias and assure overall quality, the Critical Appraisal Skills Programme (CASP) and the Mixed Methods Appraisal Tool (MMAT) were utilized [26, 27].”
MethodPhysician Empathy in Doctor-Patient Communication: A Systematic Review
Zhang, X., Li, L., Zhang, Q., Le, L. H., & Wu, Y. (2023) · Health Communication · doi:10.1080/10410236.2023.2201735 · Cited by 119
“The two researchers independently assessed the quality of the included studies using the Mixed Method Appraisal Tool (MMAT) to appraise the quality of empirical research (Hong et al., 2018).”
MethodThe Impact of AI Scribes on Streamlining Clinical Documentation: A Systematic Review
Sasseville, M., Yousefi, F., Ouellet, S., Naye, F., Stefan, T., Carnovale, V., Bergeron, F., Ling, L., Gheorghiu, B., Hagens, S., et al. (2025) · Healthcare · doi:10.3390/healthcare13121447 · Cited by 122
Williamson, O., Swann, C., Bennett, K. J. M., Bird, M. D., Goddard, S. G., Schweickle, M. J., & Jackman, P. C. (2024) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2022.2116723 · Cited by 79
“The mixed methods appraisal tool (MMAT; Hong et al., 2018; See Appendix 5) was used to assess study quality.”
Lawes-Wickwar, S., Ghio, D., Tang, M. Y., Keyworth, C., Stanescu, S., Westbrook, J., Jenkinson, E., Kassianos, A. P., Scanlan, D., Garnett, N., et al. (2021) · Vaccines · doi:10.3390/vaccines9020072 · Cited by 91
Chua, B. W. B., Ma, V., Asjes, C., Lim, A. S., Mohseni, M. M., & Wee, H. L. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18094586 · Cited by 93
“The Mixed Methods Appraisal tool (MMAT) was used to conduct a quality assessment of the included studies [14].”
MethodHeart Failure Caregiver Burden and Outcomes: A Systematic Review
Suksatan, W., Tankumpuan, T., & Davidson, P. M. (2022) · Journal of Primary Care & Community Health · doi:10.1177/21501319221112584 · Cited by 102
Davidson, A. R., Kelly, J. T., Ball, L., Morgan, M. A. J., & Reidlinger, D. P. (2022) · BMC Family Practice · doi:10.1186/s12875-021-01595-6 · Cited by 106
“Critical appraisal of the data was conducted by two independent researchers using the Mixed Methods Appraisal Tool (MMAT), version 2018 [24].”
MethodZhang, S. X., Chen, R. Z., Xu, W., Yin, A., Dong, R. K., Chen, B. Z., Delios, A. Y., Miller, S., McIntyre, R. S., Ye, W., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19021018 · Cited by 78
“Using the Mixed Methods Appraisal Tool (MMAT)18, we found that seven studies (25”
MethodGrief in family carers of people living with dementia: A systematic review
Crawley, S., Sampson, E. L., Moore, K. J., Kupeli, N., & West, E. (2022) · International Psychogeriatrics · doi:10.1017/s1041610221002787 · Cited by 119
“The Mixed Methods Appraisal Tool (MMAT) 2018 revised version with improved content validity was used to assess the quality of included studies (Hong et al. , 2018).”
Meta-analytic evidence of depression and anxiety in Eastern Europe during the COVID-19 pandemic
Zhang, S. X., Miller, S., Xu, W., Yin, A., Chen, B. Z., Delios, A., Dong, R. K., Chen, R. Z., McIntyre, R. S., Wan, X., et al. (2022) · European Journal of Psychotraumatology · doi:10.1080/20008198.2021.2000132 · Cited by 95
Baldassarre, A., Lulli, L. G., Cavallo, F., Fiorini, L., Mariniello, A., Mucci, N., & Arcangeli, G. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.1039680 · Cited by 94
“The quality assessment was performed using the Mixed Methods Appraisal Tool (MMAT) (44, 45) which is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative, and mixed methods studies, thus, it was suitable for our review.”
MethodSunlight, ultraviolet radiation, vitamin D and skin cancer: how much sunlight do we need?
Holick, M. F. (2014) · PubMed · Cited by 85
Digital Storytelling for Health-Related Outcomes in Older Adults: Systematic Review
Stargatt, J., Bhar, S. S., Bhowmik, J., & Mahmud, A. A. (2021) · Journal of Medical Internet Research · doi:10.2196/28113 · Cited by 90
“MCI: mild cognitive impairment MMAT: Mixed Methods Appraisal Tool PRISMA: Preferred Reporting Items for Systematic Review and Meta-Analyses PRISMA-P: Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols Edited by R Kukafka; submitted 22.02.21; peer-reviewed by P Pluye, A Hashim, B Hattink; comm…”
BackgroundMethodInfluentialGill, F. J., Hashem, Z., Stegmann, R., & Aoun, S. M. (2021) · Palliative Medicine · doi:10.1177/0269216320967593 · Cited by 73
Promoting Physical Activity Through Conversational Agents: Mixed Methods Systematic Review
Luo, T. C., Aguilera, A., Lyles, C. R., & Figueroa, C. A. (2021) · Journal of Medical Internet Research · doi:10.2196/25486 · Cited by 68
“Assigning studies an overall numerical score based on the ratings of each criterion is discouraged because a single number cannot provide insight into which aspects of the study methodology are problematic [26].”
Groves, S., Lascelles, K. M., & Hawton, K. E. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.03.027 · Cited by 97
A scoping review of heart rate variability in sport and exercise psychology
Mosley, E., & Laborde, S. (2022) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2022.2092884 · Cited by 96
Ulupınar, F., & Erden, Y. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16588 · Cited by 78
Evaluating Generative AI in Mental Health: Systematic Review of Capabilities and Limitations
Wang, L., Bhanushali, T., Huang, Z., Yang, J., Badami, S., & Hightow-Weidman, L. (2025) · JMIR Mental Health · doi:10.2196/70014 · Cited by 73
“We used the Mixed Methods Appraisal Tool (MMAT; version 2018)to evaluate the methodological quality of studies included in this review [19].”
Omenka, C. O., Tseja-Akinrin, A., Sen, P., Mac-Seing, M., Agbaje, A., Menzies, D., & Zarowsky, C. (2020) · BMJ Global Health · doi:10.1136/bmjgh-2019-002280 · Cited by 98
Incorporating oral health care education in undergraduate nursing curricula - a systematic review
Bhagat, V., Hoang, H., Crocombe, L., & Goldberg, L. R. (2020) · BMC Nursing · doi:10.1186/s12912-020-00454-6 · Cited by 102
“Yes Yes Yes Yes -** Overall quality score* 1.0 0.6 1.0 0.8 -** *Overall quality score: Studies met all assessment criteria scored one and studies met fewer criteria scored less than one **According to MMAT, further appraisal may not be feasible or appropriate when answer is “‘No’ or ‘Cannot ‘tell’ to screening questio…”
MethodInfluentialPatient-Centered Care for Patients with Depression or Anxiety Disorder: An Integrative Review
de Pinho, L. G., Lopes, M. J., Correia, T., Sampaio, F., do Arco, H. M. D. S. L. R., Mendes, A. Q., Marques, M. D. C., & da Fonseca, C. J. V. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11080776 · Cited by 95
“In addition, the evaluation carried out was only qualitative, without obtaining a score, as suggested by the authors who developed the tool [26].”
MethodPappa, S., Chen, J., Barnett, J., Chang, A., Dong, R. K., Xu, W., Yin, A., Chen, B. Z., Delios, A., Chen, R. Z., et al. (2021) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13306 · Cited by 89
“The Mixed Methods Appraisal Tool (MMAT)20 of seven questions was used to assess the quality of research papers included in the meta-analyses.”
BackgroundMethodInfluentialTelemedicine in surgical care in low- and middle-income countries: a scoping review
Owolabi, E. O., Mac Quene, T., Louw, J., Davies, J. I., & Chu, K. M. (2022) · World Journal of Surgery · doi:10.1007/s00268-022-06549-2 · Cited by 70
“This validated tool is used to assess the methodological quality of interventional, observational, and qualitative studies, paying specific attention to the study objectives, design, sampling, data collection, results, and study limitations [23].”
MethodFrazer, K., Mitchell, L., Stokes, D., Lacey, E., Crowley, E., & Kelleher, C. C. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047012 · Cited by 70
Coping, wellbeing, and psychopathology during high-risk pregnancy: A systematic review
Williamson, S. P., Moffitt, R. L., Broadbent, J., Neumann, D. L., & Hamblin, P. S. (2022) · Midwifery · doi:10.1016/j.midw.2022.103556 · Cited by 105
Kenny, A. J., Dickson‐Swift, V., McKenna, L., et al. (2021) · Nurse Education Today · doi:10.1016/j.nedt.2021.104860 · Cited by 69
Zapf, H., Boettcher, J., Haukeland, Y. B., Orm, S., Coslar, S., & Fjermestad, K. W. (2023) · JCPP Advances · doi:10.1002/jcv2.12205 · Cited by 111
“Since the reviewed articles comprised methodologically diverse studies, we used the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) for quality assessment.”
MethodForde‐Johnston, C., Butcher, D., & Aveyard, H. (2022) · Journal of Advanced Nursing · doi:10.1111/jan.15484 · Cited by 108
“The Critical Appraisal Skills Programme (CASP) (CASP, 2022) checklist was used for appraising the methodological quality of qualitative studies (n = 3), whereas the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) was used for quantitative, and mixed methods studies (n = 5).”
MethodLi, L., Yu, F., & Zhang, E. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100217 · Cited by 133
Dombestein, H., Norheim, A., & Husebø, A. M. L. (2019) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.12735 · Cited by 101
Vaccination uptake amongst older adults from minority ethnic backgrounds: A systematic review
Bhanu, C., Gopal, D. P., Walters, K. R., & Chaudhry, U. A. R. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003826 · Cited by 69
Alshammari, F., Sim, J., Lapkin, S., & Stephens, M. (2022) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103294 · Cited by 87
Factors Associated with Professional Mental Help-Seeking Among U.S. Immigrants: A Systematic Review
Mohammadifirouzeh, M., Oh, K. M., Basnyat, I., & Gimm, G. W. (2023) · Journal of Immigrant and Minority Health · doi:10.1007/s10903-023-01475-4 · Cited by 90
Digital dashboards visualizing public health data: a systematic review
Schulze, A., Brand, F., Geppert, J., & Böl, G. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.999958 · Cited by 95
“Studies that had executed a user study (n = 18), meaning an empirical assessment of a focal dashboard through different user groups, were considered for further quality assessment by means of the Mixed Methods Appraisal Tool (MMAT) which was specifically developed for critically appraising the quality of different stu…”
MethodLalor, J., Sheaf, G., Mulligan, A., Ohaja, M., Clive, A., Tighe, S. M., Ng, E. D., & Shorey, S. (2022) · Women and Birth · doi:10.1016/j.wombi.2022.08.004 · Cited by 71
McLean, C. P., Utpala, R., & Sharp, G. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.926709 · Cited by 130
“The methodological quality of studies was assessed using the Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018).”
MethodTighe, S. A., Ball, K., Kensing, F., Kayser, L., Rawstorn, J. C., & Maddison, R. (2020) · Journal of Medical Internet Research · doi:10.2196/16774 · Cited by 86
Giddings, R., Joseph, A., Callender, T., Janes, S. M., van der Schaar, M., Sheringham, J., & Navani, N. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00241-8 · Cited by 80
“Integration with relevant CASP criteria was therefore performed, with efforts to facilitate consistency across study types, resulting in five criteria per study type and five additional questions for mixed methods studies maintained as per MMAT (appendix pp 11–20).”
BackgroundMethodInfluentialRodríguez-Ortiz, M. Á., Santana-Mancilla, P. C., & Anido, L. (2025) · Applied Sciences · doi:10.3390/app15158679 · Cited by 170
Vallone, F., Lemmo, D., Martino, M. L., Donizzetti, A. R., Freda, M. F., Palumbo, F., Lorenzo, E., D’Argenzio, A., & Caso, D. (2022) · Psycho-Oncology · doi:10.1002/pon.5997 · Cited by 65
“The Mixed‐Methods Appraisal Tool (MMAT) version‐2018 was used.(20) This tool was developed for evaluating the methodological quality of empirical studies (qualitative/quantitative/”
MethodOsborn, T. G., Weatherburn, P., & French, R. (2021) · Journal of Adolescence · doi:10.1016/j.adolescence.2021.09.007 · Cited by 79
Vella, E., WHITE, V. M., & Livingston, P. M. (2022) · Nurse Education Today · doi:10.1016/j.nedt.2022.105500 · Cited by 68
Bozkurt, S. A., Aydoğan, S., Dursun Ergezen, F., & Türkoğlu, A. (2025) · International Nursing Review · doi:10.1111/inr.70018 · Cited by 63
Evrard, P., Pétein, C., Beuscart, J.-B., & Spinewine, A. (2022) · Implementation Science · doi:10.1186/s13012-022-01206-7 · Cited by 62
“The quality of the included studies was independently assessed by two reviewers (PE and CP), using the Mixed-Methods Appraisal Tool (MMAT), version 2018 [31].”
MethodReview of the Delphi method in library and information science research
Lund, B. D. (2020) · Journal of Documentation · doi:10.1108/jd-09-2019-0178 · Cited by 92
Geddis‐Regan, A., Errington, L., Abley, C., Wassall, R. R., Exley, C., & Thomson, R. (2020) · Health Expectations · doi:10.1111/hex.13167 · Cited by 67
Hoel, V., Feunou, C. M., & Wolf‐Ostermann, K. (2021) · BMC Geriatrics · doi:10.1186/s12877-021-02105-0 · Cited by 69
Chan, R. J., Agbejule, O. A., Yates, P. M., Emery, J. D., Jefford, M., Koczwara, B., Hart, N. H., Crichton, M., & Nekhlyudov, L. (2021) · Journal of Cancer Survivorship · doi:10.1007/s11764-021-01018-6 · Cited by 75
“” Mixed-methods studies were appraised using the mixed-methods appraisal tool [37].”
MethodRedsell, S. A., Slater, V., Rose, J., Olander, E. K., & Matvienko‐Sikar, K. (2021) · Obesity Reviews · doi:10.1111/obr.13228 · Cited by 74
Infant and young child feeding during natural disasters: A systematic integrative literature review
Mudiyanselage, S. R., Davis, D. L., Kurz, E., & Atchan, M. (2022) · Women and Birth · doi:10.1016/j.wombi.2021.12.006 · Cited by 89
HIV, Tuberculosis, and Food Insecurity in Africa—A Syndemics-Based Scoping Review
Ojo, T., Ruan, C., Hameed, T., Malburg, C., Thunga, S., Smith, J. A., Vieira, D. L., Snyder, A., Tampubolon, S., Gyamfi, J., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19031101 · Cited by 64
“The Mixed Methods Appraisal Tool (MMAT) version 2018 was utilized to assess mixed methods studies [39].”
MethodHeaney, S., Tomlinson, M., & Aventin, Á. (2022) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-022-04770-4 · Cited by 85
“In line with MMAT guidelines, two reviewers (SH and ÁA) were independently involved in the appraisal process (Hong et al., 2018).”
MethodInfluentialErwin, J., Paisi, M., Neill, S., Burns, L., Vassallo, I., Nelder, A., Facenfield, J., Devalia, U., Vassallo, T., & Witton, R. (2022) · Health Expectations · doi:10.1111/hex.13544 · Cited by 62
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.