Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool ( MMAT )
Open accessAbstract
Rationale, aims, and objectives: Systematic reviews combining qualitative, quantitative, and/or mixed methods studies are increasingly popular because of their potential for addressing complex interventions and phenomena, specifically for assessing and improving clinical practice. A major challenge encountered with this type of review is the appraisal of the quality of individual studies given the heterogeneity of the study designs. The Mixed Methods Appraisal Tool (MMAT) was developed to help overcome this challenge. The aim of this study was to explore the usefulness of the MMAT by seeking the views and experiences of researchers who have used it. Methods: We conducted a qualitative descriptive study using semistructured interviews with MMAT users. A purposeful sample was drawn from the researchers who had previously contacted the developer of the MMAT, and those who have published a systematic review for which they had used the MMAT. All interviews were transcribed verbatim and analyzed by 2 coders using thematic analysis. Results: Twenty participants from 8 countries were interviewed. Thirteen themes were identified and grouped into the 2 dimensions of usefulness, ie, utility and usability. The themes related to utility concerned the coverage, completeness, flexibility, and other utilities of the tool. Those regarding usability were related to the learnability, efficiency, satisfaction, and errors that could be made due to difficulties understanding or selecting the items to appraise. Conclusions: On the basis of the results of this study, we make several recommendations for improving the MMAT. This will contribute to greater usefulness of the MMAT.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ultimately, 84 studies with MMAT scores ≥ 3 were retained for analysis.
“Ultimately, 84 studies with MMAT(Hong et al., 2018) [60] scores ≥ 3 were retained for analysis (Figure 3).”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [60]
Usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool.
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or have contacted the developer of the tool for questions or permission to use the tool (18).”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [18]
The results of the usefulness testing were helpful to identify changes to be made in the MMAT.
“The results of this study were helpful to identify changes to be made in the MMAT.”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [18]
The Mixed Methods Appraisal Tool (MMAT) was applied to appraise quality of included studies.
“To rigorously appraise the quality of the included studies, we applied the Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018), which is suitable for evaluating qualitative, quantitative, and mixed-methods research.”
From Digital Transformation in International Trade: Opportunities, Challenges, and Policy Implications (2025) · cites this paper as Hong et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Fereday, J., & Muir-Cochrane, E. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500107 · Cited by 11,829 · see citing works
The integrative review: updated methodology
Whittemore, R., & Knafl, K. (2005) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2005.03621.x · Cited by 10,241 · see citing works
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,533 · see citing works
What's in a name? Qualitative description revisited
Sandelowski, M. (2010) · Research in Nursing & Health · doi:10.1002/nur.20362 · Cited by 4,975 · see citing works
What is an adequate sample size? Operationalising data saturation for theory-based interview studies
Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., & Grimshaw, J. M. (2010) · Psychology & Health · doi:10.1080/08870440903194015 · Cited by 3,429 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Content validity in psychological assessment: A functional approach to concepts and methods.
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995) · Psychological Assessment · doi:10.1037//1040-3590.7.3.238 · Cited by 2,387 · see citing works
Qualitative description – the poor cousin of health research?
Neergaard, M. A., Olesen, F., Andersen, R. S., & Søndergaard, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-52 · Cited by 2,099 · see citing works
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531 · see citing works
Pluye, P., & Hong, Q. N. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182440 · Cited by 1,363 · see citing works
Self-citationPace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331 · see citing works
Self-citationPluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119 · see citing works
Self-citationSystems to rate the strength of scientific evidence.
West, S. L., King, V., Carey, T. S., Lohr, K. N., Mckoy, N. A., Sutton, S. F., & Lux, L. J. (2002) · PubMed Central · Cited by 873 · see citing works
Hong, Q. N., Pluye, P., Bujold, M., & Wassef, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0454-2 · Cited by 754 · cited 1× · see citing works
“The first label refers to combining qualitative, quantitative, and/or mixed methods studies, while the second one can also refer to combining qualitative and quantitative methods (such as thematic synthesis and meta‐analysis).(7) Hereinafter, we will use the term “systematic mixed studies reviews” to designate this ty…”
MethodSelf-citationSimera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-24 · Cited by 590 · see citing works
What Is Ecological Validity? A Dimensional Analysis
Schmuckler, M. A. (2001) · Infancy · doi:10.1207/s15327078in0204_02 · Cited by 561 · see citing works
A Dictionary of Epidemiology (DRAFT)
(2014) · doi:10.1093/acref/9780199976720.001.0001 · Cited by 530 · see citing works
Introduction to systematic reviews
O’Connor, E., Whitlock, E. P., & Spring, B. J. (2011) · PsycEXTRA Dataset · doi:10.1037/e657462011-001 · Cited by 463 · see citing works
A systematic review of the content of critical appraisal tools
Katrak, P., Bialocerkowski, A. E., Massy‐Westropp, N., Kumar, S., & Grimmer, K. A. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-22 · Cited by 389 · see citing works
Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2014.08.010 · Cited by 375 · see citing works
Self-citationSandelowski, M., & Barroso, J. V. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100107 · Cited by 331 · see citing works
Mixed methods research synthesis: definition, framework, and potential
Heyvaert, M., Maes, B., & Onghena, P. (2011) · Quality & Quantity · doi:10.1007/s11135-011-9538-6 · Cited by 262 · see citing works
A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · see citing works
Carroll, C., Booth, A., & Lloyd-Jones, M. (2012) · Qualitative Health Research · doi:10.1177/1049732312452937 · Cited by 248 · see citing works
Critical Appraisal of Mixed Methods Studies
Heyvaert, M., Hannes, K., Maes, B., & Onghena, P. (2013) · Journal of Mixed Methods Research · doi:10.1177/1558689813479449 · Cited by 166 · see citing works
Santiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. L. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.007 · Cited by 146 · see citing works
A proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121 · see citing works
Mhaskar, R., Djulbegovic, B., Magazin, A., Soares, H. P., & Kumar, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.016 · Cited by 102 · see citing works
External Validity and Model Validity: A Conceptual Approach for Systematic Review Methodology
Khorsan, R., & Crawford, C. C. (2014) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2014/694804 · Cited by 96 · see citing works
Seo, H.-J., Kim, S. Y., Lee, Y. J., Jang, B.-H., Park, J.-E., Sheen, S.-S., & Hahn, S. K. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.013 · Cited by 74 · see citing works
Hartling, L., Bond, K., Santaguida, P. L., Viswanathan, M., & Dryden, D. M. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.010 · Cited by 69 · see citing works
How to Appraise the Studies: An Introduction to Assessing Study Quality
Petticrew, M. P., & Roberts, H. M. (2006) · Wiley eBooks · doi:10.1002/9780470754887.ch5 · Cited by 65 · see citing works
Barker, J. (2013) · SAGE Publications Ltd eBooks · doi:10.4135/9781529681413.n7 · Cited by 26 · see citing works
Verhage, A., & Boels, D. (2016) · Quality & Quantity · doi:10.1007/s11135-016-0345-y · Cited by 24 · see citing works
Including systematic reviews in PhD programmes and candidatures in nursing - 'Hobson's choice'?
Olsson, C., Ringnér, A., & Borglin, G. (2014) · Nurse Education in Practice · doi:10.1016/j.nepr.2014.01.005 · Cited by 21 · see citing works
Heffernan, E., Oving, I., Barry, T., Phung, V.-H., Siriwardena, A., & Masterson, S. (2019) · HRB Open Research · doi:10.12688/hrbopenres.12969.2 · Cited by 7 · see citing works
Risk of bias assessment should not go beyond reporting assessment
Faggion, C. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.11.014 · Cited by 5 · see citing works
Flagged first, then by citations
1 of the 40 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Dixon‐Woods M, 2004, Integrative approaches to qualitative and quantitative evidence
- Heyvaert M, 2016, Using Mixed Methods Research Synthesis for Literature Reviews: The Mixed Methods Research Synthesis Approach
- Bai A, 2012, Quality Assessment Tools Project Report
- Pluye P, Proposal: A mixed methods appraisal tool for systematic mixed studies reviews
- 10.1037/1040-3590.7.3.238 · DOI · Paper page
Cited by
Sorted by citations
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737 · see citing works
“To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a).”
Self-citationImproving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.008 · Cited by 1,082 · see citing works
“In addition, in interviews conducted with MMAT users to explore their views and experiences of the MMAT, concerns were raised about whether the tool included enough criteria to judge the quality of studies and criteria that were difficult to judge, in particular the criteria for qualitative and mixed methods studies […”
MethodSelf-citationMethodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2020) · JBI Evidence Synthesis · doi:10.11124/jbisrir-d-19-00169 · Cited by 747 · see citing works
Noyes, J., Booth, A., Moore, G., Flemming, K., Tunçalp, Ö., & Shakibazadeh, E. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-000893 · Cited by 497 · see citing works
Ressel, M., Thompson, B., Poulin, M.-H., Normand, C. L., Fisher, M. H., Couture, G., & Iarocci, G. (2020) · Autism · doi:10.1177/1362361320910963 · Cited by 388 · see citing works
Primary medical care continuity and patient mortality: a systematic review
Baker, R., Freeman, G. K., Haggerty, J. L., Bankart, M. J., & Nockels, K. H. (2020) · British Journal of General Practice · doi:10.3399/bjgp20x712289 · Cited by 331 · see citing works
Hu, D., Zhou, S., Crowley‐McHattan, Z. J., & Liu, Z. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18063147 · Cited by 342 · see citing works
Zamboni, K., Baker, U., Tyagi, M., Schellenberg, J., Hill, Z. E., & Hanson, C. (2020) · Implementation Science · doi:10.1186/s13012-020-0978-z · Cited by 231 · see citing works
Methodological guidance for the conduct of mixed methods systematic reviews
Stern, C., Lizarondo, L. M., Carrier, J., Godfrey, C. M., Rieger, K. L., Salmond, S. W., Apóstolo, J., Kirkpatrick, P., & Loveday, H. P. (2021) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000282 · Cited by 214 · see citing works
An integrative review of leadership competencies and attributes in advanced nursing practice
Heinen, M. M., van Oostveen, C., Peters, J. W. B., Vermeulen, H., & Huis, A. M. P. (2019) · Journal of Advanced Nursing · doi:10.1111/jan.14092 · Cited by 208 · see citing works
Kaboré, S. S., Ngangue, P., Soubeiga, D., Barro, A., Pilabré, A. H., Bationo, N., Pafadnam, Y., Drabo, K. M., Hien, H., & Savadogo, L. G. B. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1014375 · Cited by 223 · see citing works
Fong, V. C., & Iarocci, G. (2020) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsaa092 · Cited by 206 · see citing works
Organisational factors and non-accidental violence in sport: A systematic review
Roberts, V. L., Sojo, V. E., & Grant, F. (2019) · Sport Management Review · doi:10.1016/j.smr.2019.03.001 · Cited by 176 · see citing works
Women’s autonomy in healthcare decision making: a systematic review
Idris, I. B., Hamis, A. A., Bukhori, A. B. M., Hoong, D. C. C., Yusop, H., Shaharuddin, M. A.-A., Fauzi, N. A. F. A., & Kandayah, T. (2023) · BMC Women s Health · doi:10.1186/s12905-023-02792-4 · Cited by 176 · see citing works
Nudging toward vaccination: a systematic review
Reñosa, M. D. C., Landicho, J., Wachinger, J., Dalglish, S. L., Bärnighausen, K., Bärnighausen, T., & McMahon, S. A. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-006237 · Cited by 170 · see citing works
Huo, Y., Couzner, L., Windsor, T. D., Laver, K., Dissanayaka, N. N., & Cations, M. (2023) · Implementation Science Communications · doi:10.1186/s43058-023-00428-0 · Cited by 171 · see citing works
Pallavicini, F., Pepe, A., & Mantovani, F. (2022) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2021.0252 · Cited by 168 · see citing works
Tovin, M. M., & Wormley, M. E. (2023) · Physical Therapy · doi:10.1093/ptj/pzad084 · Cited by 153 · see citing works
Wong, E., Mavondo, F., & Fisher, J. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-05383-3 · Cited by 146 · see citing works
Chomutare, T., Tejedor, M., Svenning, T. O., Marco-Ruiz, L., Tayefi, M., Lind, K., Godtliebsen, F., Moen, A., Ismail, L., Makhlysheva, A., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192316359 · Cited by 156 · see citing works
Rapid reviews: the pros and cons of an accelerated review process
Moons, P., Goossens, E., & Thompson, D. R. (2021) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvab041 · Cited by 139 · see citing works
Little, M., Rosa, E., Heasley, C., Asif, A., Dodd, W., & Richter, A. (2021) · American Journal of Health Promotion · doi:10.1177/08901171211056584 · Cited by 136 · see citing works
A review of virtual-simulation for assessing healthcare students' clinical competency
Coyne, E., Calleja, P., Forster, E., & Lin, F. F. (2020) · Nurse Education Today · doi:10.1016/j.nedt.2020.104623 · Cited by 135 · see citing works
Yu, C., Sommerlad, A., Sakure, L., & Livingston, G. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101633 · Cited by 131 · see citing works
“We used the Mixed Method Appraisal Tool (MMAT) version 2018 (Hong et al., 2018) to assess the study quality of each included study.”
MethodMental Health during the COVID-19 Crisis in Africa: A Systematic Review and Meta-Analysis
Chen, J., Chen, J., Zhang, M., Chen, R. Z., Kechen, D. R., Dong, Z., Ye, Y., Tong, L., Chen, B. Z., Zhao, R., et al. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182010604 · Cited by 124 · see citing works
Froehlich, D. E., Waes, S. V., & Schäfer, H. (2020) · Review of Research in Education · doi:10.3102/0091732x20903311 · Cited by 141 · see citing works
Pimentel, J., Ansari, U., Omer, K., Gidado, Y., Baba, M. C., Andersson, N., & Cockcroft, A. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-2852-z · Cited by 124 · see citing works
Ghio, D., Lawes-Wickwar, S., Tang, M. Y., Epton, T., Howlett, N., Jenkinson, E., Stanescu, S., Westbrook, J., Kassianos, A. P., Watson, D., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-048750 · Cited by 123 · see citing works
Commercial Off-The-Shelf Video Games for Reducing Stress and Anxiety: Systematic Review
Pallavicini, F., Pepe, A., & Mantovani, F. (2021) · JMIR Mental Health · doi:10.2196/28150 · Cited by 127 · see citing works
Sclar, G. D., Penakalapati, G., Caruso, B. A., Rehfuess, E. A., Garn, J. V., Alexander, K. T., Freeman, M. C., Boisson, S., Medlicott, K., & Clasen, T. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.09.016 · Cited by 113 · see citing works
The Impact of Nurse Leadership Styles on Nurse Burnout:
Wei, H., King, A. T., Jiang, Y., Sewell, K. B., & Lake, D. (2020) · Nurse Leader · doi:10.1016/j.mnl.2020.04.002 · Cited by 112 · see citing works
Rahimi, A. K., Pienaar, O., Ghadimi, M., Canfell, O. J., Pole, J. D., Shrapnel, S., van der Vegt, A. H., & Sullivan, C. M. (2024) · Journal of Medical Internet Research · doi:10.2196/49655 · Cited by 131 · see citing works
Interprofessional Collaboration in Long-Term Care and Rehabilitation: A Systematic Review
Doornebosch, A. J., Smaling, H. J. A., & Achterberg, W. P. (2022) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2021.12.028 · Cited by 105 · see citing works
Renwick, L., Pedley, R., Johnston, I., Bell, V., Lovell, K., Bee, P., & Brooks, H. L. (2022) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-022-01997-6 · Cited by 121 · see citing works
The role of digital health technology in rural cancer care delivery: A systematic review
Morris, B. B., Rossi, B., & Fuemmeler, B. (2022) · The Journal of Rural Health · doi:10.1111/jrh.12619 · Cited by 104 · see citing works
Khosravipour, M., Khanlari, P., Khazaie, S., Khosravipour, H., & Khazaie, H. (2021) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2021.101427 · Cited by 103 · see citing works
Campbell, A. R., Layne, D. M., Scott, E. S., & Wei, H. (2020) · Journal of Nursing Management · doi:10.1111/jonm.13083 · Cited by 101 · see citing works
Acknowledging cancer as a family disease: A systematic review of family care in the cancer setting.
Coyne, E., Heynsbergh, N., & Dieperink, K. (2020) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2020.101841 · Cited by 90 · see citing works
Burnett, A. M., Yashadhana, A., Lee, L., Serova, N., Brain, D., & Naidoo, K. S. (2018) · Bulletin of the World Health Organization · doi:10.2471/blt.18.212332 · Cited by 97 · see citing works
Gribben, L., & Semple, C. J. (2020) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2020.101887 · Cited by 103 · see citing works
Yopa, D. S., Massom, D. M., Kiki, G. M., Sophie, R. W., Sylvie, F., Thiam, O., Zinaba, L., & Ngangue, P. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1252428 · Cited by 95 · see citing works
Hudon, C., Chouinard, M., Pluye, P., Sherif, R. E., Bush, P. L., Rihoux, B., Poitras, M.-È., Lambert, M., Zomahoun, H. T. V., & Légaré, F. (2019) · The Annals of Family Medicine · doi:10.1370/afm.2419 · Cited by 91 · see citing works
Self-citationPatterns and motivations of polysubstance use: a rapid review of the qualitative evidence
Boileau-Falardeau, M., Contreras, G. A., Gariépy, G., & Laprise, C. (2022) · Health Promotion and Chronic Disease Prevention in Canada · doi:10.24095/hpcdp.42.2.01 · Cited by 90 · see citing works
Making change last? Exploring the value of sustainability approaches in healthcare: a scoping review
Lennox, L., Linwood-Amor, A., Maher, L., & Reed, J. E. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-00601-0 · Cited by 94 · see citing works
Social Media Use and Deliberate Self-Harm Among Youth: A Systematized Narrative Review
Biernesser, C., Sewall, C. J. R., Brent, D., Bear, T., Mair, C., & Trauth, J. (2020) · Children and Youth Services Review · doi:10.1016/j.childyouth.2020.105054 · Cited by 86 · see citing works
Yang, Y., Gao, Y., An, R., & Wan, Q. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104808 · Cited by 95 · see citing works
McBride, E., Tatar, O., Rosberger, Z., Rockliffe, L., Marlow, L. A. V., Moss-Morris, R. E., Kaur, N., Wade, K., & Waller, J. (2020) · Health Psychology Review · doi:10.1080/17437199.2020.1762106 · Cited by 85 · see citing works
Establishing a clinical service to prevent psychosis: What, how and when? Systematic review
de Pablo, G. S., Estradé, A., Cutroni, M., Andlauer, O., & Fusar‐Poli, P. (2021) · Translational Psychiatry · doi:10.1038/s41398-020-01165-x · Cited by 83 · see citing works
Chee, R. M., Capper, T. S., & Muurlink, O. T. (2023) · Midwifery · doi:10.1016/j.midw.2023.103623 · Cited by 81 · see citing works
Synthesis Methods and Reporting Tool (SMART) for Research Syntheses in Applied Linguistics
Chong, S. W. (2025) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2025.2456880 · Cited by 79 · see citing works
Health system readiness for non-communicable diseases at the primary care level: a systematic review
Kabir, A., Karim, M. N., Islam, R. M., Romero, L., & Billah, B. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060387 · Cited by 77 · see citing works
Moyle, W., Murfield, J., & Lion, K. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104513 · Cited by 75 · see citing works
Meyer, D., Slone, S., Ogungbe, O., Duroseau, B., & Farley, J. E. (2022) · AIDS and Behavior · doi:10.1007/s10461-022-03771-w · Cited by 74 · see citing works
Moral distress among acute mental health nurses: A systematic review
Lamoureux, S., Mitchell, A. E., & Forster, E. (2024) · Nursing Ethics · doi:10.1177/09697330241238337 · Cited by 72 · see citing works
Characteristics that influence purchase choice for cannabis products: a systematic review
Donnan, J. R., Shogan, O., Bishop, L. D., Swab, M., & Najafizada, M. (2022) · Journal of Cannabis Research · doi:10.1186/s42238-022-00117-0 · Cited by 71 · see citing works
Wen-hong, Z., Jones, C., Wu, M., & Moyle, W. (2020) · Journal of Clinical Nursing · doi:10.1111/jocn.15451 · Cited by 70 · see citing works
Singh, N., Kovacs, R., Cassidy, R., Kristensen, S. R., Borghi, J., & Brown, G. W. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113624 · Cited by 70 · see citing works
Factors underpinning unsafe driving: A systematic literature review of car drivers
Bowen, L., Budden, S. L., & Smith, A. P. (2020) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2020.04.008 · Cited by 68 · see citing works
Giddings, R., Joseph, A., Callender, T., Janes, S. M., van der Schaar, M., Sheringham, J., & Navani, N. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00241-8 · Cited by 70 · see citing works
Robert, R. S., Stavinoha, P. L., Jones, B. L., Robinson, J. E., Larson, K., Hicklen, R., Smith, B. W. H., Perko, K., Koch, K. D., Findley, S., et al. (2019) · Pediatric Blood & Cancer · doi:10.1002/pbc.27764 · Cited by 63 · see citing works
Renwick, L., Pedley, R., Johnston, I., Bell, V., Lovell, K., Bee, P., & Brooks, H. L. (2021) · Health Expectations · doi:10.1111/hex.13407 · Cited by 75 · see citing works
Henderson, D., Sewell, K. B., & Wei, H. (2019) · International Journal of Nursing Sciences · doi:10.1016/j.ijnss.2019.12.009 · Cited by 61 · see citing works
The utility of physiotherapy assessments delivered by telehealth: A systematic review
Zischke, C., Simas, V., Hing, W. A., Milne, N., Spittle, A. J., & Pope, R. (2021) · Journal of Global Health · doi:10.7189/jogh.11.04072 · Cited by 63 · see citing works
Patient-facing digital tools for delivering genetic services: a systematic review
Lee, W., Shickh, S., Assamad, D., Luca, S., Clausen, M., Somerville, C., Tafler, A., Shaw, A., Hayeems, R. Z., & Bombard, Y. (2022) · Journal of Medical Genetics · doi:10.1136/jmg-2022-108653 · Cited by 60 · see citing works
The implementation of the nursing process in lower‐income countries: An integrative review
Lotfi, M., Zamanzadeh, V., Valizadeh, L., Khajehgoodari, M., Rezaei, M. E., & Khalilzad, M. A. (2019) · Nursing Open · doi:10.1002/nop2.410 · Cited by 61 · see citing works
Brew‐Sam, N., Chhabra, M., Parkinson, A., Hannan, K., Brown, E., Pedley, L., Brown, K., Wright, K., Pedley, E., Nolan, C. J., et al. (2021) · JMIR Diabetes · doi:10.2196/20973 · Cited by 58 · see citing works
Parmar, J., Sacrey, L. A., Anderson, S., Charles, L., Dobbs, B., McGhan, G., Shapkin, K., Tian, P., & Triscott, J. (2022) · Health & Social Care in the Community · doi:10.1111/hsc.13578 · Cited by 58 · see citing works
Factors Influencing Social Isolation among Cancer Patients: A Systematic Review
Wang, C., Qiu, X., Yang, X., Mao, J., & Li, Q. (2024) · Healthcare · doi:10.3390/healthcare12101042 · Cited by 66 · see citing works
Li, J., Li, Y., Hu, Y., Ma, C. F., Mei, X., Chan, E. A., & Yorke, J. (2025) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.70059 · Cited by 64 · see citing works
Savage, M., Savage, C., Brommels, M., & Mazzocato, P. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-035542 · Cited by 57 · see citing works
Effects of mHealth on the psychosocial health of pregnant women and mothers: a systematic review
Sakamoto, J. L., Carandang, R. R., Kharel, M., Shibanuma, A., Львовна, Я. Е., Александровна, Б. М., & Jimba, M. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-056807 · Cited by 61 · see citing works
Relationships and Mechanisms Between Occupational Risk Factors and Distal Upper Extremity Disorders
Keir, P. J., Farias Zuniga, A., Mulla, D. M., & Somasundram, K. G. (2021) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720819860683 · Cited by 57 · see citing works
Saif‐Ur‐Rahman, K., Islam, M. S., Alaboson, J., Ola, O., Hasan, I., Islam, N., Mainali, S., Martina, T., Silenga, E., Muyangana, M., et al. (2023) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12547 · Cited by 56 · see citing works
Stigma in functional neurological disorder (FND) – A systematic review
McLoughlin, C., McWhirter, L., Pisegna, K., Tijssen, M. A. J., Tak, L. M., Carson, A. J., & Stone, J. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102460 · Cited by 56 · see citing works
Resilience and mental health among perinatal women: a systematic review
Hajure, M., Alemu, S. S., Abdu, Z., Mulatu, G., Alemayehu, Y., Dule, A., Hussen, M. A., Wedajo, L. F., & Gezimu, W. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1373083 · Cited by 56 · see citing works
Willis, S., Neil, R., Mellick, M., & Wasley, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00393 · Cited by 54 · see citing works
Piat, M., Wainwright, M., Sofouli, E., Vachon, B., Deslauriers, T., Préfontaine, C., & Frati, F. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01646-0 · Cited by 55 · see citing works
Context, Input, Process, and Product Evaluation Model in medical education: A systematic review
Toosi, M., Modarres, M., Amini, M., & Geranmayeh, M. (2021) · Journal of Education and Health Promotion · doi:10.4103/jehp.jehp_1115_20 · Cited by 55 · see citing works
Duran, G., Uysal‐Bozkir, Ö., Uittenbroek, R. J., van Hout, H. P. J., & Groenou, M. I. B. V. (2021) · Dementia · doi:10.1177/14713012211055307 · Cited by 55 · see citing works
Virtual Reality Applications in Medicine During the COVID-19 Pandemic: Systematic Review
Pallavicini, F., Pepe, A., Clerici, M., & Mantovani, F. (2022) · JMIR Serious Games · doi:10.2196/35000 · Cited by 60 · see citing works
Jones, C., Jones, D., & Moro, C. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053616 · Cited by 54 · see citing works
Huddlestone, L., Shoesmith, E., Pervin, J., Lorencatto, F., Watson, J., & Ratschen, E. (2022) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntac004 · Cited by 54 · see citing works
Peer support groups after acquired brain injury: a systematic review
Hughes, R. L., Fleming, P., & Henshall, L. (2020) · Brain Injury · doi:10.1080/02699052.2020.1762002 · Cited by 52 · see citing works
Women’s health-related vulnerabilities in natural disasters: a systematic review protocol
Fatema, S. R., Islam, M. S., East, L., & Usher, K. J. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-032079 · Cited by 52 · see citing works
Paley, C. A., Boland, J. W., Santarelli, M., Murtagh, F. E., Ziegler, L. E., & Chapman, E. (2023) · BMC Palliative Care · doi:10.1186/s12904-023-01202-8 · Cited by 50 · see citing works
Telemedicine in the primary care of older adults: a systematic mixed studies review
Ilali, M., Berre, M. L., Vedel, I., & Khanassov, V. (2023) · BMC Family Practice · doi:10.1186/s12875-023-02085-7 · Cited by 50 · see citing works
Laprise, C., & Bolster‐Foucault, C. (2021) · Canada Communicable Disease Report · doi:10.14745/ccdr.v47i02a03 · Cited by 49 · see citing works
O’Donnell, S., Lohan, M., Oliffe, J. L., Grant, D., & Galway, K. (2021) · Arts & Health · doi:10.1080/17533015.2021.1894463 · Cited by 50 · see citing works
Aronsson, J., Nichols, A., Warwick, P., & Elf, M. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15950 · Cited by 49 · see citing works
Hodgson, C. R., Mehra, R., & Franck, L. S. (2024) · Children · doi:10.3390/children11080949 · Cited by 49 · see citing works
Determinants of Self-Care and Home-Based Management of Hypertension: An Integrative Review
Konlan, K. D., & Shin, J. (2023) · Global Heart · doi:10.5334/gh.1190 · Cited by 79 · see citing works
The impact of storytelling on building resilience in children: A systematic review
Ramamurthy, C., Zuo, P., Armstrong, G., & Andriessen, K. (2023) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.13008 · Cited by 48 · see citing works
Finn, M., Walsh, A., Rafter, N., Mellon, L., Chong, H. Y., Naji, A., O'Brien, N., Williams, D. J., & McCarthy, S. E. (2024) · BMJ Open Quality · doi:10.1136/bmjoq-2023-002506 · Cited by 48 · see citing works
Use of academic electronic medical records in nurse education: A scoping review
Raghunathan, K., McKenna, L., & Peddle, M. (2021) · Nurse Education Today · doi:10.1016/j.nedt.2021.104889 · Cited by 47 · see citing works
Arora, A. K., Ortiz-Paredes, D., Engler, K., Lessard, D., Mate, K., Rodriguez-Cruz, A., Kronfli, N., Vedel, I., Cox, J., Quesnel‐Vallée, A., et al. (2021) · AIDS Patient Care and STDs · doi:10.1089/apc.2021.0079 · Cited by 46 · see citing works
Balancing hospital governance: A systematic review of 15 years of empirical research
Regge, M. D., & Eeckloo, K. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113252 · Cited by 46 · see citing works
Gunn, V., Kreshpaj, B., Santander, N. M., Vignola, E. F., Wegman, D. H., Hogstedt, C., Ahonen, E. Q., Bodin, T., Orellana, C., Baron, S., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042232 · Cited by 49 · see citing works
Partnering with frail or seriously ill patients in research: a systematic review
Ludwig, C., Graham, I. D., Gifford, W., Lavoie, J. G., & Stacey, D. (2020) · Research Involvement and Engagement · doi:10.1186/s40900-020-00225-2 · Cited by 45 · see citing works
Xia, Q., Liu, Q., Tlili, A., & Chiu, T. K. F. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105465 · Cited by 67 · see citing works
Digital Transformation in International Trade: Opportunities, Challenges, and Policy Implications
Mirzaye, S., & Mohiuddin, M. (2025) · Journal of Risk and Financial Management · doi:10.3390/jrfm18080421 · Cited by 43 · see citing works
- Authors
- Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P.
- Affiliations
- McGill University (CA)
- Department of Family Medicine McGill University Montréal QC Canada
- Corresponding author
- Pluye, Pierre
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2018-02-21
- Volume
- 24
- Issue
- 3
- Pages
- 459-467
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2794014169
- Cited by
- 1,107
- References
- 40
- Citations by year
- 2026: 227 · 2025: 230 · 2024: 176 · 2023: 156 · 2022: 130 · 2021: 78 · 2020: 84 · 2019: 20 · 2018: 5
- FWCI
- 34.23
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- Journal of Evaluation in Clinical Practice (publishedVersion)
- PubMed (publishedVersion)
- eScholarship@McGill (McGill) (submittedVersion)
- Funders
- Canadian Institutes of Health Research (301011)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- Mixed Methods Appraisal Tool, mixed methods, study quality appraisal, systematic reviews, tool usability, researcher experience, complex interventions
- MeSH terms
- Female, Humans, Interviews as Topic, Male, Research Design, Qualitative Research, Internationality, Quality Improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
