The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.
Abstract
INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.
Overview
CoChat Summary
TL;DR
This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.
Key findings
The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.
Methods
This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.
Limitations
The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Abstract reports semi-structured interviews with 20 participants to explore the MMAT’s usefulness. The Study Participants section says researchers were recruited through systematic reviews that used the MMAT and a list of people who had contacted its developer for permission or guidance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results of this study were helpful to identify changes to be made in the MMAT.”
What the sources say
- Supportedcites [18] 18. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459-67. source
The Discussion says the study explored MMAT users’ experiences to help revise the tool and identified themes suggesting areas for improvement. The Recommendations for Improving the MMAT then proposes specific changes on the basis of those results. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (7).”
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-46. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (21).”
What the sources say
- Not matchedcites [21] 21. Pluye P. Critical appraisal tools for assessing the methodological quality of qualitative, quantitative and mixed methods studies included in systematic mixed studies reviews. J Eval Clin Pract. 2013;19(4):722.
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
5 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Quality was assessed using two validated tools by study design. Quantitative studies were appraised using the Mixed Methods Appraisal Tool Version 2018, which evaluates five methodological criteria on a binary scale (Hong et al., 2018). Qualitative studies were additionally appraised using the Critical Appraisal Skills Programme Qualitative Checklist, which evaluates 10 criteria (CASP UK, 2018).”
From Cultural stigma and caregiver burden among family caregivers of people with nervous system disorders in the Middle East and North Africa: A mixed-methods systematic review (2026) · cites this paper as Hong et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was assessed using the 2018 version of the Mixed Methods Appraisal Tool (MMAT).15”
From Psychosocial interventions for siblings of children with cancer: A mixed methods systematic review (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“To further limit risks of bias, a formal risk of bias assessment tool (ROB-tool), the Mixed Methods Appraisal Tool [52], was used to evaluate the overall study quality.”
From Mediators of outcome in adolescent psychotherapy and their implications for theories and mechanisms of change: a systematic review (2024) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“LM, with the support of a research assistant, assessed the methodological quality of empirical studies (qualitative, quantitative and mixed methods) using the Mixed Methods Appraisal Tool (MMAT) [45].”
From Transnationalism and care of migrant families during pregnancy, postpartum and early-childhood: an integrative review (2020) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“The quality of the included studies was independently assessed by two authors using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).Disagreements were resolved through discussion.”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Hong et al., 2018
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Conclusion · p. 5
“## 4. Conclusion”
§ 4. How to use the MMAT? · p. 4
“## 4. How to use the MMAT?”
Citations
Cites
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · cited 2×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · cited 3×
“This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009).”
InfluentialSelf-citationHong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107 · cited 3×
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 923
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464 · cited 1×
“The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000).”
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · cited 1×
“Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015).”
Self-citationA review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · cited 2×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · cited 1×
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Adjustment of meta-analyses on the basis of quality scores should be abandoned
Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.03.008 · Cited by 207 · cited 1×
“Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012).”
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 168 · cited 1×
“Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016).”
Cited by
Items 3,301–3,400 of 4,578
Carbuccia, L. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/2f98s
Kelly, Y., O’Rourke, N., Flynn, R., et al. (2021) · HRB Open Research · doi:10.12688/hrbopenres.13212.1
Pitt, E., Bradford, N., Robertson, E. G., Sansom‐Daly, U. M., & Alexander, K. E. (2022) · Research Square · doi:10.21203/rs.3.rs-1480070/v1
Szymkowicz, E., Bodet-Contentin, L., Marechal, Y., & Ehrmann, S. (2022) · Research Square · doi:10.21203/rs.3.rs-1644919/v1
Singh, S., & Kapoor, S. (2022) · Evidence-Based Nursing · doi:10.1136/ebnurs-2022-103539
AlSwayied, G., Guo, H., Rookes, T., Frost, R., & Hamilton, F. L. (2022) · doi:10.2196/preprints.40271
Visit Types in Primary Care with Telehealth Use during COVID-19: Systematic Review (Preprint)Preprint
Ward, K., Vagholkar, S., Sakur, F., Khatri, N. N., & Lau, A. Y. S. (2022) · doi:10.2196/preprints.40469
Brobbin, E., Deluca, P., Hemrage, S., & Drummond, C. (2022) · doi:10.2196/preprints.40210
Intervenciones para el manejo psicoemocional de pacientes con Covid 19: una revisión exploratoria
Ortiz, A. G., Jeraldo, J. C., & Cea, X. (2022) · Revista Chilena de Atención Primaria y Salud Familiar · doi:10.5354/2735-7805.2022.67503
“utilizó la Herramienta de Evaluación para Métodos Mixtos (MMAT por sus siglas en inglés), (14) la cual está diseñada precisamente para la etapa de evaluación de revisiones sistemáticas de estudios mixtos, que permitió valorar la calidad metodológica a través de cinco grandes categorías: investigación cualitativa, ensa…”
BackgroundPeng, Y., Wu, T., Chen, Z., & Deng, Z. (2022) · Journal of Medical Internet Research · doi:10.2196/39397
“This tool not only appraises the quality of individual studies given the heterogeneity of the study designs but also accounts for many biases including confounding bias, nonresponse bias, and sampling bias [4].”
Eaton, M. M. (2021) · Digital Repository Service (Northeastern University) · doi:10.17760/d20423504
Вус, В., Nakanishi, Y., & da Silva, G. T. B. (2022) · Wiadomości Lekarskie · doi:10.36740/wlek202208102
Al‐Masaeed, M. (2022) · Japan Journal of Clinical & Medical Research · doi:10.47363/jjcmr/2022(2)126
Baker, M., Griffin, A., Dacre, J., & Medisauskaite, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-062356
Cubelo, F., Parviainen, A., Turunen, H. E., et al. (2022) · doi:10.37766/inplasy2022.11.0075
Family experiences during illness outbreaks: A systematic reviewPreprint
Sheen, J. A., Chen, L., Lee, B., Aridas, A., & Reupert, A. (2022) · medRxiv · doi:10.1101/2022.11.16.22282428
A Systematic Review of the Risk and Protective Factors of Tobacco Use among South Indian Adults
Daniel, M., Rajkumar, E., George, A. J., & Romate, J. (2022) · The Open Public Health Journal · doi:10.2174/18749445-v15-e221020-2022-70 · Cited by 2
Protocol for mixed-method systematic review of urology in medical school education
Samaranayake, D., Pattenden, T. A., Morton, A., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-061418
Chandy, P., Criel, B., & Lakshmi, L. (2022) · Manipal Journal of Nursing and Health Sciences · doi:10.55889/2582-7979.1257
Ravalihasy, A., Ante-Testard, P. A., Kardaś-Słoma, L., Yazdanpanah, Y., Allegri, M. D., & Ridde, V. (2023) · AIDS and Behavior · doi:10.1007/s10461-023-04000-8 · Cited by 1
Exploring the conduct and reporting of qualitative research in trials using mixed methods
Clement, C. (2023) · Cronfa (Swansea University) · doi:10.23889/suthesis.62588
An Integrated Study of Parent Satisfaction with Ongoing Home Visitation for Moms and Children
Myhydeen, Y. M. (2023) · Journal of Biology and Life Science · doi:10.5296/jbls.v14i1.20759
Saga, M., Rkhaila, A., & Ounine, K. (2022) · Advances in public policy and administration (APPA) book series · doi:10.4018/978-1-6684-6496-0.ch003 · Cited by 2
Research Use in Postsecondary Education and Workforce TrainingPreprint
Chuter, C., & Shewchuk, S. (2023) · doi:10.35542/osf.io/gqkb5
Fitzharris, L., McGowan, E., & Broderick, J. M. (2023) · HRB Open Research · doi:10.12688/hrbopenres.13671.1 · Cited by 6
“This approach also aligns with the typology as developed by Hong et al .”
Santonja-Ayuso, L., Portillo, M. C., & Corchón, S. (2023) · Research Square · doi:10.21203/rs.3.rs-2350805/v1
A systematic review of nursing theory-based delirium studies
Kim, S. H., Kim, H., Kim, Y., & Lee, K. H. (2023) · Science Talks · doi:10.1016/j.sctalk.2023.100202
Burton, H., & Gordon, K. (2023) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.13095 · Cited by 1
Zhuang, S., Tan, D. W., Reddrop, S., Dean, L., Maybery, M. T., & Magiati, I. (2023) · doi:10.31219/osf.io/cgzbu
Комунікація без заподіяння шкоди: оглядове дослідження висвітлення в медіа травматичних подій
Onufriv, S., Abdelhamid, O., & van Dormolen, S. (2023) · doi:10.30970/vjo.2023.52-53.11747
Morrison, N. M. V., Morrison, B. W., & Mahlberg, J. (2023) · Research Square · doi:10.21203/rs.3.rs-2749986/v1 · Cited by 1
Sofianingrum, A., Jati, S. P., & Martini, M. (2023) · Jurnal Keperawatan Komprehensif · doi:10.33755/jkk.v9i2.513
“MMAT uses a threshold value of 80 to determine whether the selected articles are of high quality or not (9,10).”
Effects of Equine-Assisted Therapy on Recovery after Stroke – A Systematic ReviewPreprint
Trunk, B. H., & Gharabaghi, A. (2023) · medRxiv · doi:10.1101/2023.05.09.23289710
Stronczek, M. K., Cruel, E., Steiner, L., et al. (2023) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2023.03.006
Dillarstone, H., Brown, L. J., & Flores, E. C. (2023) · doi:10.31223/x54m30
Nxele, S. R., Moetlhoa, B., Dlangalala, T., et al. (2023) · medRxiv · doi:10.1101/2023.05.28.23290656
Pragastiwi, E. A., & Pratiwi, C. S. (2023) · Jurnal Aisyah Jurnal Ilmu Kesehatan · doi:10.30604/jika.v8i2.1937
“Penelitian ini mengguanakan PRISMA-ScR dengan frame work PEO, 3 data base dan 1 grey literatur, artikel diseleksi melalui Prisma Flowchart dan Critical Appraisal menggunakan Joanna Briggs Institute (JBI) dan Mixed Method Appraisal Tool (MMAT).”
MethodInfluentialAlikeju, E. H., & Samuel, A. S. (2023) · International journal of research and scientific innovation · doi:10.51244/ijrsi.2023.10515
ACE-2 receptor-associated COVID-19 lesions and sequelae
Flores, M. A. B., Santos-Castañeda, A., Benítez-Hernández, A. K., Molina-Trinidad, E. M., & Hernández-Tapia, K. (2023) · Mexican Journal of Medical Research ICSA · doi:10.29057/mjmr.v11i22.10559
“The following inclusion criteria were used: No publication time restriction, Spanish, English, and Portuguese languages, systematic reviews, metaanalyses, and case reports, in addition to the reference lists of those articles; a total of 87 publications were selected that had our keywords in both the title and abstrac…”
Hoong, D. C. C., Ismail, H., & Ahmad, H. (2023) · East African Scholars Journal of Medical Sciences · doi:10.36349/easms.2023.v06i07.001
“However, this method does not apply to assessing reviews and theoretical papers (Hong et al., 2018).”
Methodde Campos, A. C., Hidalgo‐Robles, Á., Longo, E., Shrader, C., & Paleg, G. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15717
Exercise Effect on Mental Health in Isolating or Quarantining Adults
Chu, V., & Newman, D. G. (2023) · Aerospace Medicine and Human Performance · doi:10.3357/amhp.6073.2023
Santos, J. D. C., Arreguy-Sena, C., Pinto, P. F., et al. (2023) · Revista gaúcha de enfermagem · doi:10.1590/1983-1447.2023.20220170.en
“The description followed the investigation criteria of the Mixed Methods Appraisal Tool (MMAT), version 2018 (19) .”
MethodAcessibilidade arquitetônica e percepção de quedas de idosos no peridomicílio: método misto
Santos, J. D. C., Arreguy-Sena, C., Pinto, P. F., et al. (2023) · Revista gaúcha de enfermagem · doi:10.1590/1983-1447.2023.20220170.pt
Resource integration in the electric vehicle ecosystem from a user perspective in Australia
Alotaibi, A. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.242727
Gender Disparity of Tuberculosis Burden in Low-and Middle-Income Countries: A Systematic Review
Phyoe, N., Hlaing, T., & Sandar, W. P. (2023) · Health Informatics - An International Journal · doi:10.5121/hiij.2023.12302
“The score was provided as 20 % to 100% depending on the meeting of criteria [54].”
de Campos, A. C., Hidalgo‐Robles, Á., Longo, E., Shrader, C., & Paleg, G. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15756
Factors Determining Turnaround Strategies for Declining, Large, High-tech organizations
Jeyasingh, B. B. F. (2023) · Journal of Humanities and Social Sciences Studies · doi:10.32996/jhsss.2023.5.10.2
“The combined MMAT+TAPUPASM framework assigned a scale of three for each criterion, with the highest possible score being 39 and the lowest possible score being zero.”
MethodFactors Determining Turnaround Strategies for Declining, Large, High-tech organizations
Jeyasingh, B. B. F. (2023) · Journal of Business and Management Studies · doi:10.32996/jhsss.2023.5.5.16
Almadadha, L., Gholizadeh, L., & Sheppard‐Law, S. (2023) · Advances in health sciences research/Advances in Health Sciences Research · doi:10.2991/978-94-6463-248-4_7
Farraj, N. H., Toosi, A., Qahtani, A. A., Forgrave, D., & Ansar, S. (2023) · International Journal of Healthcare · doi:10.5430/ijh.v9n2p8 · Cited by 1
The online testing and evaluation in the higher education
Tran, T. T. T. (2022) · Science & Technology Development Journal - Social Sciences & Humanities · doi:10.32508/stdjssh.v6i2.737
Mixed Methods Research in Criminology
Brent, J. J., Kraska, P. B., & Skeen, J. (2023) · Oxford University Press eBooks · doi:10.1093/obo/9780195396607-0330 · Cited by 2
Hamato, O. H., Johnson, J. M., Arbabi, H., Forgrave, D., & Ansar, S. (2023) · International Journal of Healthcare · doi:10.5430/ijh.v9n2p34
Kinch, M., Kroll, T., Stokes, D., & Somanadhan, S. (2023) · HRB Open Research · doi:10.12688/hrbopenres.13684.3
Centenaro, A. P. F. C., da Silva, R. M., Franco, G. P., et al. (2023) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.6669.4001
Kerstens, S., Warmer, J., Ziylan, C., & Kuijt-Evers, L. F. M. (2023) · Springer series in design and innovation · doi:10.1007/978-3-031-32198-6_23
Body Weight Discrimination Against Women in Customer-Facing Roles: A Systematic Literature Review
Grant, S., Mizzi, T., & O’Loghlen, E. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80071-174-720230005
Formulating a code of ethics for Kuwaiti pharmacists using FIP ethical standards: A Delphi study
Alfailakawi, R., Waheedi, S., & Al‐Haqan, A. (2023) · International Journal of Pharmacy Practice · doi:10.1093/ijpp/riad074.004
A Systematic Review of Non-Police Response Models for Persons in Crisis
Brook, G. (2023) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2023-15672
(2023) · Frontiers research topics · doi:10.3389/978-2-8325-4084-8
Simms-Moore, R., Dures, E. K., Barua, N. U., & Cramp, F. L. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-078183
“42 Collating, summarising and reporting Gaps will be highlighted and where appropriate recommendations for future research and clinical practice will be identified.”
BackgroundINTERVENÇÃO EDUCATIVA COM ENFERMEIROS SOBRE CONSULTA DE PUERICULTURA: UM ESTUDO DE MÉTODO MISTO
Vieira, D. D. S., Soares, A. R., Guedes, A. T. A., et al. (2023) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-0132pt
Okwor, C. J., Okwor, V. C., Meka, I. A., Emedoh, A. E., & Nweke, M. (2023) · Medical Principles and Practice · doi:10.1159/000535965
“The mixed-method appraisal tool (MMAT) Version 2011 was utilized [28].”
MethodMaternity care for refugees and asylum seekers in the Netherlands
Verschuuren, A. E. H. (2024) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.856543803 · Cited by 1
Duma, N. E., Benjamin‐Damons, N., Pefile, N., & Hlongwana, K. W. (2024) · doi:10.2196/preprints.56724
Webber, M., Moran, N., & Naughton‐Doe, R. (2024) · Research on Social Work Practice · doi:10.1177/10497315241230667
“Where studies used a combination of cross-sectional surveys and qualitative methods, an approach recommended by Hong et al. (2018) was utilized, that involved fi rst assessing each component of the study (in this case, we used CASP and AXIS), before then using a Mixed Methods Appraisal Tool (MMAT)…”
Likoti, P. G., Kuupiel, D., & Khuzwayo, N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0289515
Fredriksson, M., Hoel, T., & Jernes, M. (2023) · Nordic Journal of Systematic Reviews in Education · doi:10.23865/njsre.v1.5530 · Cited by 1
“The MMAT is a tool that will appraise the research quality of different types of studies, such as qualitative research, quantitative descriptive studies, and mixed methods studies in the same review (Hong et al., 2018b).”
MethodAtkinson-Graham, M., Brunton, G., Cancelliere, C., Corso, M., de Zoete, A., Rubinstein, S. M., Murnaghan, K., & Mior, S. A. (2024) · Digital Health · doi:10.1177/20552076241236573
Child Health Nurses' Acceptance and Use of a Novel Telehealth Platform
Goudswaard, L., Penny, R. A., Edmunds, J., & Arnautovska, U. (2024) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001116
Humayra, S., Yahya, N. A., Ning, C. J., Mir, I. A., Mohamed, A. L., & Manan, H. A. (2024) · Research Square · doi:10.21203/rs.3.rs-3757447/v1
Parents' Coping Strategies for their School Children's Issues: A Systematic Literature Review
Jamaluddin, A., Isa, N. J. M., Zaidi, M. H. A., Midan, A. R., & Zulkifli, M. N. (2024) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v13-i1/21181 · Cited by 1
Cooper, J., Murphy, J., Woods, C. B., van Nassau, F., McGrath, A., Callaghan, D., Carroll, P., Kelly, P., Murphy, N. M., & MURPHY, M. H. (2021) · Figshare · doi:10.6084/m9.figshare.16584533
Shoesmith, A., Hall, A. E., Wolfenden, L., Shelton, R. C., Powell, B. J., Brown, H., McCrabb, S., Sutherland, R., Yoong, S., Lane, C., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14775643
Leadership Style and Quality of Care for Nursing: A Rapid Review
Ali, M. F. A. (2024) · Revista de Gestão Social e Ambiental · doi:10.24857/rgsa.v18n7-077
Ogungbe, O., Jabakhanji, S. B., Mehta, R., McCaffrey, J., Byrne, D., Hurley, S. M., Rosman, L. M., Bansah, E. C., Ibukun, F. A., Quarshie, I. A., et al. (2024) · medRxiv · doi:10.1101/2024.04.17.24305997
Rayner, C. L. (2024) · Cronfa (Swansea University) · doi:10.23889/suthesis.66311 · Cited by 1
Dolatabadi, Z., Moridi, M., Farnam, F., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4313573/v1
Safarifard, R., Kiernan, G., Corcoran, Y., Courtney, E., Mitchell, J., Akard, T. F., & Lambert, V. (2024) · HRB Open Research · doi:10.12688/hrbopenres.13891.1 · Cited by 2
Abdo, M., & Schlösser, A. (2024) · British Paramedic Journal · doi:10.29045/14784726.2024.6.9.1.34 · Cited by 3
“The methodological quality of each paper was assessed using the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018).”
InfluentialПраведников, А. В., Perkovic, S., & Lagerkvist, C. J. (2024) · doi:10.31219/osf.io/vd3cr
Aboyeji, A., Mah, S., Teare, G. F., & Adhikari, K. (2024) · Population Medicine · doi:10.18332/popmed/189377
School-based strategies to prevent depression in adolescents: An integrative review
AlKurbi, S. H. A., Toosi, A., Forgrave, D., & Ansar, S. (2024) · International Journal of Healthcare · doi:10.5430/ijh.v10n1p29 · Cited by 1
Workplace incivility in hospitals: a significant barrier to patient safety and care quality
Natan, M. B. (2024) · Evidence-Based Nursing · doi:10.1136/ebnurs-2024-104078
da Luz, E. M. F., Munhoz, O. L., Greco, P. B. T., et al. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7048.4175
Lee, H., Shin, Y., Choi, Y., Moon, H., & Lee, A. (2024) · doi:10.2196/preprints.63170
Tamborini, M. M. D. F., Colet, C. D. F., Centenaro, A. P. F. C., et al. (2024) · Cogitare Enfermagem · doi:10.1590/ce.v29i0.94736
MUSCULOSKELETAL PAIN IN PRIMARY CARE PROFESSIONALS DURING THE COVID-19 PANDEMIC: MIXED METHODS STUDY
Tamborini, M. M. D. F., Colet, C. D. F., Centenaro, A. P. F. C., et al. (2024) · Cogitare Enfermagem · doi:10.1590/ce.v29i0.94735
Tamborini, M. M. D. F., Colet, C. D. F., Centenaro, A. P. F. C., et al. (2024) · Cogitare Enfermagem · doi:10.1590/ce.v29i0.91903
Evaluation of laboratory tests for COVID-19 in South Africa.
Samsunder, N. (2023) · ResearchSpace (University of KwaZulu-Natal) · doi:10.29086/10413/23146
Acceptability and feasibility of maternal screening for Group B Streptococcus: a rapid reviewPreprint
Constantinou, G., Webb, R., Ayers, S., Mitchell, E. J., & DANIELS, J. P. (2024) · medRxiv · doi:10.1101/2024.06.28.24309381
“The quality of the included studies was assessed using: the questionnaire critical appraisal checklist 16 for quantitative questionnaire papers; the Joanna Briggs Cross-Sectional Studies tool for other quantitative papers 17 ; the Walsh and Downe 18 tool for qualitative studies and; the Mixed Methods Appraisal Tool 19…”
MethodProulx, A., Arbour, C., & Pomey, M. (2024) · Recherche en soins infirmiers · doi:10.3917/rsi.156.0018
du Toit, T. L., Soeker, M. S., & Wegner, L. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082903 · Cited by 1
“19 A quality appraisal of included studies will be conducted using the Mixed Methods Appraisal Tool 2018 version by Hong et al (2018).”
MethodDodd, S., & Sydes, M. L. (2024) · European Journal on Criminal Policy and Research · doi:10.1007/s10610-024-09596-8 · Cited by 2
“The Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) was used to assess article quality in this review since it accommodates qualitative, quantitative, and mixed methods studies within a single tool (Pace et al., 2012).”
InfluentialChu, J. T. W., Wilson, H., Cai, C. Z., McCormack, J. C., Newcombe, D. A. L., & Bullen, C. (2024) · JMIR Mental Health · doi:10.2196/51074 · Cited by 1
“The methodological quality of each study was accessed using the Mixed Methods Appraisal Tool (MMAT) [33,34].”
MethodZhao, Q., Cooke, A., Aurizki, G. E., & Dowding, D. (2024) · Studies in health technology and informatics · doi:10.3233/shti240324
Lasheen, R. A., Kavanagh, P. M., Doyle, B., Downey, C., Siersbaek, R., Stanistreet, D., & Doyle, F. (2024) · HRB Open Research · doi:10.12688/hrbopenres.13923.1 · Cited by 1
“…good fit for process-oriented intervention research in public health context 42,45 A methodological rating of “yes”, “no”, or “can’t tell” and domain descriptors will be assigned to different aspects of the included studies to produce an overall score with justifications provided 42,44,46 .”
Shaikh, M. F., Higley, C., Campanile, C., Francis, B., Panja, E., Santacaterina, S., Pratesi, G., & Piaggio, D. (2024) · doi:10.2196/preprints.64627 · Cited by 1
Chisnall, G., Kumpunen, S., Manby, L., van Susante, H., Stevens, E., Rankl, F., Robinson, H., Hoernke, K., Johnson, G. A., & Vindrola‐Padros, C. (2024) · SAGE Open · doi:10.1177/21582440241261924
“Namely, the Mixed Methods Appraisal Tool (MMAT) for research articles (Hong et al., 2018) and the AACODS for gray literature (Tyndall, 2010).”
Metadata
- Authors
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
- Affiliations
- McGill University (CA)
- Universitat Oberta de Catalunya (ES)
- University of Warwick (GB)
- University of Canberra (AU)
- Université de Sherbrooke (CA)
- Université Laval (CA)
- University of Sheffield (GB)
- Institut National de la Recherche Scientifique (CA)
- Corresponding author
- Pluye, Pierre
- ORCID
- Venue
- Education for Information
- Publisher
- SAGE Publications
- Published
- 2018-12-18
- Volume
- 34
- Issue
- 4
- Pages
- 285-291
- ISSN
- 0167-8329 (ISSN-L)
- 1875-8649
- OpenAlex
- W2901724277
- Cited by
- 4,737
- References
- 21
- Citations by year
- 2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
- FWCI
- 102.21
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Education for Information (publishedVersion)
- O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- HAL (submittedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
- EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.